Model restore failed

Warning

The AI Accelerator requires ROS 2 Humble, so PolyScope X 10.12.1 is the latest compatible release. PolyScope X 10.13 and later use ROS 2 Jazzy and are not currently supported.

When ./scripts/run_sdk_setup.sh starts the SDK container, the entrypoint restores or compiles Isaac models (SAM / Mobile SAM and FoundationPose). If that fails, Detection, Auto Annotation, Pose Estimation, and Dynamic Picking will not work correctly.

← Back to Troubleshooting

Symptoms

Summary:
------------------
 SAM: Failure
 FoundationPose: Success

 SAM (Mobile): Failure

Or at runtime:

failed to load 'segment_anything' version 1: Internal: failed to stat file
/workspaces/isaac_ros-dev/isaac_ros_assets/models/segment_anything/1/model.onnx

Also common: restore message prints, but SAM stays Failure; path planning may still work.

Cause

On first start, if SAM is missing, the entrypoint either restores from host models_backup or downloads and compiles models (GPU-heavy). Compiling while not on MAXN can leave incomplete files. Those files are copied into models_backup, so later starts keep restoring the bad backup.

Paths

Role

Host

Container

Persistent backup (bind mount)

~/aia_sdk/ros/isaac_ros_assets/models_backup/

/workspaces/isaac_ros-dev/isaac_ros_assets/models_backup/

Runtime models

Restored inside the container workspace

/workspaces/isaac_ros-dev/isaac_ros_assets/models/

SAM ONNX (required)

.../models_backup/segment_anything/1/model.onnx

.../models/segment_anything/1/model.onnx

FoundationPose engines

.../models_backup/foundationpose/*.plan

.../models/foundationpose/*.plan

Healthy backup layout (sizes vary):

~/aia_sdk/ros/isaac_ros_assets/models_backup/
├── foundationpose/       # ~250 MB — .onnx + .plan
├── segment_anything/
│   ├── config.pbtxt
│   └── 1/model.onnx      # ~40 MB — required for SAM Success
├── rtdetr/
└── resnet/

Fix

Run on the host. Do not interrupt a compile.

  1. Stop the Isaac container if it is running:

docker ps
docker stop isaac_pkgs_app

  1. Set power mode to MAXN (id 0):

sudo nvpmodel -m 0
nvpmodel -q

Expect:

NV Power Mode: MAXN
0

Other modes on this platform: MODE_15W (1), MODE_30W (2), MODE_50W (3). Use MAXN for compile and normal SDK use.

  1. Delete the bad backup so the next start recompiles:

sudo rm -rf ~/aia_sdk/ros/isaac_ros_assets/models_backup

  1. Re-run setup and wait for completion:

cd ~/aia_sdk/ros
./scripts/run_sdk_setup.sh

Success looks like:

Summary:
------------------
 SAM: Success
 FoundationPose: Success

 SAM (Mobile): Success

then SETUP COMPLETE - CONTAINER READY.

  1. Confirm from another host terminal:

ls -lh ~/aia_sdk/ros/isaac_ros_assets/models_backup/segment_anything/1/model.onnx
ls -lh ~/aia_sdk/ros/isaac_ros_assets/models_backup/foundationpose/*.plan

Still failing?

Check

Action

Not on MAXN after nvpmodel -m 0

Reboot, then set MAXN again

Compile interrupted

Delete models_backup and re-run setup

Disk full

df -h ~ — free space under data/ (Clean up data files)

SAM Failure, FoundationPose Success

Missing/bad segment_anything/1/model.onnx — rebuild on MAXN

Note

Only delete models_backup when you intend a full rebuild. After a good setup it is copied so later starts restore quickly.