How PandaI uses NVIDIA Isaac

Warning

The AI Accelerator requires ROS 2 Humble, so PolyScope X 10.12.1 is the latest compatible release. PolyScope X 10.13 and later use ROS 2 Jazzy and are not currently supported.

Why read this guide?

This page is the mental model for the AI Accelerator (PandaI): what Universal Robots owns, what NVIDIA Isaac ROS runs on the Jetson GPU, and how a PolyScope program gets a part pose or a collision-aware path. Read it before the examples if you need to know why the launch files start so many nodes.

PandaI is the software name for the perception and planning stack that ships with the UR AI Accelerator. On disk and in ROS it is pandai_ark (the ARK stack). NVIDIA Isaac ROS / Isaac Manipulator supplies the GPU models and planners. UR supplies the camera/calibration URCapX, the Docker workspace, the glue nodes, and the ark_* URScript API.

PandaI is built as detect → pose → plan → move.

The robot never loads a .onnx file itself. A PolyScope program calls ROS 2 services. Those services run inside the SDK container on the NVIDIA Jetson Orin AGX 64GB.

What runs where

Layer

Software

Machine

Teach pendant / robot OS

PolyScope X 10.12.1, URCaps (aia_core, camera, aia-sdk)

UR controller

Camera control and hand-eye

AIA Core, Camera Creator

Jetson (containers) + pendant UI

RGB-D driver

Orbbec Gemini 335Lg ROS 2 node

Jetson

GPU AI + planning

NVIDIA Isaac ROS (RT-DETR, SAM/Triton, FoundationPose, cuMotion)

Jetson, container isaac_pkgs_app

Glue, world model, URScript

pandai_ark (pipeline_image_node, world_model_updater, ur_cumotion)

Same SDK container

Host workspace: ~/aia_sdk/ros/
Inside the container: /workspaces/isaac_ros-dev/
Start / enter the container: cd ~/aia_sdk/ros then ./scripts/run_sdk_setup.sh (see SDK).

        flowchart LR
    cam["Orbbec Gemini 335Lg<br/>RGB + depth"]
    isaac["<b>NVIDIA Isaac ROS</b><br/>RT-DETR · SAM · FoundationPose · cuMotion"]
    ark["<b>pandai_ark</b><br/>pipeline + world model + URScript services"]
    psx["PolyScope X<br/>ark_infer_* / ark_move_path_plan"]

    cam --> isaac
    isaac --> ark
    ark --> psx

    classDef n fill:#EDF1F2,stroke:#002B39,stroke-width:1.5px,color:#002B39
    class cam,isaac,ark,psx n

NVIDIA pieces PandaI actually uses

These are Isaac / Isaac Manipulator building blocks, compiled to TensorRT on first SDK setup (MAXN power mode — see Jetson compute utilities and Model restore failed).

Isaac capability

What it does in PandaI

Typical assets

RT-DETR

2D object detection (bounding boxes + class)

Fine-tuned checkpoint under data/models/rtdetr_active/; base weights in isaac_ros_assets/models/rtdetr/

SAM (Segment Anything, Triton)

Turns each box into a pixel mask

ONNX / engine under isaac_ros_assets/models/segment_anything/

FoundationPose

6DOF pose from RGB-D + mask + textured mesh

score_trt_engine.plan, refine_trt_engine.plan; your .obj + texture

cuMotion

Collision-aware motion planning

Collision YAML (data/collision_*.yaml), robot model from serial / type in config.yaml

Isaac ROS camera / NITROS

GPU-friendly image types between Isaac nodes

Used inside SAM / FoundationPose launches

Classification and AprilTags are ARK nodes, not FoundationPose. Classification trains a small ONNX classifier on cropped ROIs. Tags use a tag detector and TF, not a CAD mesh.

PandaI is built as detect → pose → plan → move. It is not a vision–language–action policy (Groot / π₀). Those would be a separate inference service, not ark_load_detection_model.

UR glue on top of Isaac

pandai_ark does not replace Isaac. It orchestrates Isaac and talks to PolyScope.

ARK piece

Role

pipeline_image_node

Freezes a synced RGB-D frame; calls RT-DETR; for pose estimation prompts SAM then FoundationPose (pipeline_type "fp")

rtdetr_node

Loads your checkpoint; infer_detection2d / load_detection_model

world_model_updater

Tracks parts in the cell after 2D or 3D detection

ur_cumotion

Wraps cuMotion as /ur_cumotion/plan_motion for ark_move_path_plan

Overlay, Console UI, logging

Desktop and pendant visualization

ur-ros2-web-bridge

Relays ROS to the pendant

PolyScope calls (after ark_init()), documented under URScript API:

URScript

Isaac-side effect

ark_load_detection_model("rtdetr_active/checkpoint-…")

Load RT-DETR weights (/workspaces/isaac_ros-dev/data/models/ prefix)

ark_set_detection_params / ark_infer_detection

Trigger + wait; with FoundationPose context this is a 3D pose, not only a 2D box

ark_load_classification_model / ark_infer_classification

Crop ROI, run classifier

ark_detect_tags

Tag pipeline + TF

ark_path_planning_init / ark_move_path_plan

cuMotion plan, then the robot executes the trajectory

Perception pipeline (Isaac in the loop)

Used by Detection, Pose Estimation, and Dynamic Picking.

        flowchart TB
    rgb["RGB + depth<br/>Camera Creator topics"]
    det["<b>RT-DETR</b><br/>2D boxes"]
    sam["<b>SAM</b><br/>mask from box"]
    fp["<b>FoundationPose</b><br/>6DOF vs textured mesh"]
    wait["wait_detection<br/>ark_infer_detection"]
    move["PolyScope move_frame / pick"]

    rgb --> det
    det --> sam
    sam --> fp
    fp --> wait
    wait --> move

    classDef i fill:#FFFFFF,stroke:#002B39,stroke-width:1.5px,color:#002B39
    class rgb,det,sam,fp,wait,move i

  1. Camera Creator publishes color/image_raw, depth, and camera_info (after AIA Core has selected and started the Gemini).

  2. Hand-eye from AIA Core (/etc/aix_cameras/camera.yaml) puts those poses in robot base.

  3. RT-DETR needs a COCO-trained detector. Train it with Auto Annotation (SAM2 helps label video; training still produces RT-DETR checkpoints).

  4. SAM is a frozen Isaac model. You do not train SAM for a new part.

  5. FoundationPose does not read COCO. It matches live RGB-D + mask to a dimensionally accurate textured mesh (centered .obj + one texture image). Weak texture or a wrong mesh scale shows up as bad orientation, not as a missing box.

  6. The robot uses the returned pose (poses[0] in base) plus a taught grasp offset.

Path planning (basic / advanced) is a second Isaac path: cuMotion uses the UR model + collision YAML. Dynamic picking runs both pose estimation and cuMotion.

Training vs Isaac factory models

You train / provide

Isaac restores / compiles (do not delete)

RT-DETR checkpoint (data/models/rtdetr_active/checkpoint-*)

isaac_ros_assets/models/rtdetr/ base

Classification ONNX (data/models/classification_active/)

SAM ONNX + TensorRT

Textured part mesh (data/meshes/…)

FoundationPose *.plan engines

Collision file + gripper STL

cuMotion / robot description

First run_sdk_setup.sh restore/compile is GPU-heavy. If it fails, later starts keep a bad models_backup — see Model restore failed.

What is not NVIDIA Isaac

Keep these separate when debugging:

  • PolyScope X and URCaps — robot OS, ark_* preamble, application nodes.

  • AIA Core — camera pick, exposure, chessboard hand-eye. No RT-DETR.

  • Camera Creator — Orbbec lifecycle. If Viewer or ROS cannot see the camera, fix this (and udev / USB vs GMSL) before blaming Isaac.

  • Orbbec Viewer — host GUI, not ROS. Stop isaac_pkgs_app while using it.

Only one SDK session should own the GPU: docker stop isaac_pkgs_app before starting a second example or Viewer (Existing container not stopped).

How an example maps to Isaac

Each v2 script is ros2 launch launch/aia_example_*.launch.py inside the container.

Example

Isaac used

What you prepare

AprilTags

(tag detector; no FP)

Printed 36h11 tags

Classification

— (ARK classifier)

ROI + labeled crops

Detection

RT-DETR

COCO / auto-annotation

Pose estimation

RT-DETR + SAM + FoundationPose

Detector + textured mesh

Path planning

cuMotion

Collision YAML

Dynamic picking

All of the above

Pose assets + collision file

Launch files include Isaac launches such as isaac_ros_segment_anything_triton.launch.py and isaac_ros_foundationpose.launch.py. ARK nodes (rtdetr_node, pipeline_image_node) sit beside them.