NVIDIA Isaac ecosystem research notes

Goal: summarize the current NVIDIA Isaac ecosystem — Isaac Gym, Isaac Sim, Isaac Lab, Isaac ROS, and drone-adjacent tooling — for indoor autonomous drone mapping/capture.

Core take: Isaac Gym is legacy, Isaac Sim is robotics simulation, Isaac Lab is robot-learning on top, Isaac ROS is deployment/perception middleware. For indoor drone work that means Isaac Gym only for old-paper reproduction, Isaac Sim/Pegasus/PX4 for drone simulation, and Isaac Lab only for narrow learned components.

Ecosystem map

Component Current role Use it for Avoid using it for
Isaac Gym Deprecated/legacy GPU RL simulator Reproducing old Gym/GLEAM/IsaacGymEnvs work New projects, ROS, realistic sensors, drone integration
Isaac Sim General robotics simulator on Omniverse Sensors, ROS 2, synthetic data, SIL/HIL-style validation, high-fidelity scenes Massive RL training as the only abstraction
Isaac Lab Open-source robot-learning framework on Isaac Sim RL/IL, policy training, parallel envs, domain randomization Full autonomy stack, SLAM/capture ledger/product orchestration
Isaac ROS ROS 2 GPU-accelerated robotics packages Visual SLAM, nvblox mapping, image/depth acceleration on Jetson Offline learning benchmarks
Pegasus Simulator Drone/multirotor framework on Isaac Sim PX4 SITL, multirotor dynamics, drone-specific Isaac workflow Broad robot-learning framework

Isaac Lab replaces the older IsaacGymEnvs, OmniIsaacGymEnvs, and Orbit lineage. Isaac Sim is the substrate. Isaac ROS is closer to deployment/runtime robotics.

Isaac Gym

Status: deprecated legacy software. NVIDIA still lets developers download it, but it is no longer supported and NVIDIA recommends Isaac Lab.

What it was good at:

  • GPU-accelerated PhysX RL simulation.
  • Tensor API for large batches of environments.
  • URDF/MJCF import.
  • Fast policy training for manipulation/locomotion-style tasks.

Limitations now matter:

  • Not a general robotics simulator.
  • No serious ROS workflow.
  • No high-fidelity RTX sensor/rendering stack.
  • Old Python/CUDA/PyTorch constraints make reproducibility brittle.

Implication: a pinned Isaac Gym Preview 4 environment is worth keeping for GLEAM reproduction only. New drone work should not start on Gym.

Isaac Sim

Status: current NVIDIA general robotics simulator. Public docs list Isaac Sim 6.0.1 GA as current release line.

Capabilities relevant to indoor drone autonomy:

  • Omniverse/USD scene graph and asset pipeline.
  • PhysX plus experimental Newton support.
  • RTX cameras, depth, lidar, radar, ultrasonic-style sensors.
  • ROS 2 bridge: images, camera info, TF, odometry, joint states, point clouds, laser scans, simulation control.
  • Synthetic data generation and domain randomization.
  • URDF/MJCF/USD robot import and robot setup tools.
  • Headless/standalone simulation modes.

Drone-specific caveat: Isaac Sim itself is not a full drone stack. It gives physics/sensors/ROS/runtime. For PX4-style multirotor sim, add Pegasus or another aerial robotics layer.

Isaac Lab

Status: current NVIDIA robot-learning framework on Isaac Sim. Latest releases observed: 3.0 beta line for Isaac Sim 6.0.x; 2.3.x is the last older mainline release.

Core capabilities:

  • RL and imitation-learning task framework.
  • GPU-accelerated parallel environments.
  • Tiled rendering and vision observations.
  • Domain randomization.
  • Multi-GPU/multi-node training for RL-Games, RSL-RL, and skrl.
  • Integrations with learning libraries and manager/direct environment styles.
  • Built-in task families: classic control, manipulation, contact-rich assembly, locomotion, navigation, multirotor, Crazyflie hover, humanoid imitation.

Drone relevance:

  • Useful for training a component: local obstacle avoidance, hover/control policy, exploration heuristic, view scoring, sim-to-real robust perception-action loop.
  • Not a replacement for SLAM, ESDF mapping, coverage ledger, global planner, flight-controller failsafes, or ROS 2 integration.

Implication: if a classical frontier/NBV planner bottlenecks, a narrow Isaac Lab policy is the right response. Isaac Lab is not a system architecture.

Isaac ROS

Isaac ROS is the deployment-side NVIDIA ROS 2 stack. It matters more for real drone runtime than Isaac Lab does.

Relevant packages:

  • Isaac ROS Visual SLAM: GPU-accelerated stereo VIO/SLAM; candidate external odometry source for PX4/ArduPilot.
  • Isaac ROS nvblox: TSDF/ESDF reconstruction from depth; useful local collision map substrate.
  • Image/depth processing nodes and NVIDIA-accelerated transport.

Implication: for Jetson/VOXL-class companion autonomy, pair PX4 + ROS 2 with Isaac ROS Visual SLAM + nvblox, then run a conservative planner/safety supervisor on top.

Pegasus Simulator and drone simulation

Pegasus Simulator is a Python framework on Isaac Sim for multirotor vehicles.

Capabilities:

  • Multirotor dynamics on Isaac Sim.
  • PX4 integration.
  • Custom Python control interface.
  • ROS 2 launch support in newer versions.
  • Experimental ArduPilot integration.
  • Example workflows for extension mode and standalone applications.

Caveats:

  • Version compatibility is strict with Isaac Sim releases.
  • It is drone-specific glue, not a robot-learning framework.
  • ArduPilot support is less mature than PX4.

Implication: Pegasus + PX4 SITL is the place to test offboard setpoints, estimator/failsafe behavior, depth-camera data flow, ROS 2 bagging, and indoor scene interactions before real prop tests.

Fit to indoor autonomous drone mapping/capture

Best split:

  1. Research benchmark lane: Habitat-Sim / GLEAM / active-mapping papers. Fast iteration on embodied navigation, exploration, map completion metrics. Good for understanding policies and metrics, weak for real drone integration.
  2. Robotics simulation lane: Isaac Sim + Pegasus + PX4 SITL + ROS 2. Slower, but tests sensors, timing, offboard control, failsafes, and runtime architecture.
  3. Learning lane: Isaac Lab for narrow policies if needed. Train modules, not whole autonomy.
  4. Deployment lane: PX4 + Jetson + Isaac ROS Visual SLAM/nvblox + custom coverage ledger/planner.

Recommended sequence:

  • Keep legacy Isaac Gym env pinned for GLEAM evidence.
  • Build minimal Isaac Sim/Pegasus/PX4 scene: one room, depth camera, odometry, ROS 2 bag.
  • Implement coverage ledger + frontier/NBV planner as ROS 2 nodes.
  • Replay simulated bags into the planner before closed-loop flight.
  • Only introduce Isaac Lab after a concrete learned-module target is defined.

References