- Adds shared deployment mode enums in gr00t/deployment/modes.py and wires them into ONNX export, TRT build, verification, and benchmark CLIs.

- Tightens TRT export/build behavior: BF16-only pipeline config, static batch-size docs, ViT exported/built as FP32 while the rest stays BF16, strongly-typed precision validation, per-component precision inference from ONNX names, and full-pipeline build failure when any ONNX input/engine is missing.
- Refactors TRT runtime lifecycle: trt_torch.Engine now has explicit close() cleanup instead of an atexit hook.
- Simplifies video decoding to torchcodec-only, removing ffmpeg/decord/opencv/pyav dispatch paths, video_backend config/CLI args, av dependencies, and related obsolete tests.
- Reworks rollout video recording to stream frames through ffmpeg, stabilize caption buffer height, support explicit record_video_keys, and avoid shape changes during H.264 encoding.
- Hardens eval wrappers: env-prefix-to-embodiment mapping now fails with actionable errors, and MultiStepWrapper uses a typed AggregateMethod with fail-fast validation.
- Updates N1.7 model processing: letterbox padding is disabled by default and made optional, image transform inputs are validated more strictly, single-model tyro union parsing is fixed, and Qwen3-VL RoPE inv_freq buffers are recomputed after load for deterministic behavior.
- Adds explicit PolicyServer / PolicyClient cleanup and context-manager support, including server startup usage in run_gr00t_server.py and the DROID example client.
- Cleans up training/data code: hides modality configs from tyro CLI, removes dataset video_backend plumbing, deletes dead flowmatching/eval-accuracy code, and removes custom pytest log banners.
- Updates Docker and platform deps: global image venv under /opt/gr00t-venv, aarch64 wheel bootstrap support, Thor wheel LFS tracking, TensorRT libs pinned to NVIDIA index, dev deps for pytest-xdist/coverage/licensing, and platform pyprojects without av.
- Updates examples/docs: DROID video saving now uses imageio instead of moviepy, SimplerEnv metrics/docs are refreshed, and deployment docs remove video-backend options.
- Expands tests around Qwen3 RoPE/no-weight loading, TRT precision and missing-input handling, TRT engine lifecycle, deployment mode consistency, multistep aggregation, policy service lifecycle, and torchcodec video behavior; removes obsolete PyAV/backend-dispatch tests.

Co-authored-by: Jiajing Chen <jiajingc@nvidia.com>
Co-authored-by: Ryan Halabi <ryhalabi@nvidia.com>

id: f12a6f10fcb60bf317397fe363a50b077b88ae58

Author: SyncBot

.gitattributes

@@ -4,3 +4,4 @@ demo_data/**/*.parquet filter=lfs diff=lfs merge=lfs -text
 scripts/deployment/dgpu/wheels/*.whl filter=lfs diff=lfs merge=lfs -text
 scripts/deployment/orin/wheels/*.whl filter=lfs diff=lfs merge=lfs -text
 scripts/deployment/spark/wheels/*.whl filter=lfs diff=lfs merge=lfs -text
+scripts/deployment/thor/wheels/*.whl filter=lfs diff=lfs merge=lfs -text

.pre-commit-config.yaml

@@ -0,0 +1,25 @@
+exclude: ^(external_dependencies/)
+repos:
+  # Ruff: lint + autofix
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.12.7
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+
+  # Catch cross-platform pyproject pin drift (dGPU/Orin/Spark/Thor) at commit
+  # time; a missed mirror otherwise only surfaces at install on the unsynced
+  # platform. tomli is the 3.10 backport of stdlib tomllib (3.11+).
+  - repo: local
+    hooks:
+      - id: check-manifest-alignment
+        name: cross-platform pyproject manifest alignment
+        entry: python tools/check_manifest_alignment.py
+        language: python
+        additional_dependencies: ["tomli; python_version < '3.11'"]
+        # Mirrors MANIFESTS in tools/check_manifest_alignment.py; keep in sync when
+        # adding a platform. A miss only drops the local trigger — the unit-test
+        # live gate reads MANIFESTS directly and still fails CI on drift.
+        files: ^(pyproject\.toml|scripts/deployment/(orin|spark|thor)/pyproject\.toml|tools/(check_manifest_alignment\.py|manifest_alignment\.toml))$
+        pass_filenames: false

ATTRIBUTIONS.md

@@ -136,7 +136,7 @@ curl -LsSf https://astral.sh/uv/install.sh | sh
 
 #### dGPU (x86_64) — Default
 
-Install FFmpeg (required by `torchcodec`, the default video backend):
+Install FFmpeg (required by `torchcodec`, the only supported video backend):
 ```sh
 sudo apt-get update && sudo apt-get install -y ffmpeg
 ```
@@ -175,7 +175,7 @@ Note: GPU dependencies (flash-attn, TensorRT) may require manual installation wi
 
 > **GB300 (sm_103) Users:** Triton 3.3.1 (pinned by PyTorch 2.7) does not support the GB300 GPU architecture (sm_103). `torch.compile` will fail on GB300. Use PyTorch eager mode or TensorRT inference instead. Triton 3.5.1+ adds sm_103 support but is not yet compatible with the pinned PyTorch version.
 
-> **aarch64 Video Backend:** On aarch64 platforms (Thor, Orin, Spark), `torchcodec` is the required video backend. `install_deps.sh` prefers the prebuilt aarch64 wheel under `scripts/deployment/dgpu/wheels/` (shared by Thor/Spark against FFmpeg 6; Orin uses a matching build against FFmpeg 4) and falls back to a source build only if the wheel is missing. If you encounter `NotImplementedError` from the video backend, ensure `torchcodec` was installed successfully during setup. Other backends (decord, pyav) are not supported on aarch64.
+> **Video Backend:** GR00T uses [`torchcodec`](https://github.com/pytorch/torchcodec) as its sole video decoding backend. Backends such as `decord` and `pyav` are no longer supported. `torchcodec` requires FFmpeg and supports H.264 on all platforms; AV1 decoding is not guaranteed (convert AV1 datasets to H.264 with `examples/SimplerEnv/convert_av1_to_h264.py`). On aarch64 platforms (Thor, Orin), `torchcodec` is built from source during `install_deps.sh` because pre-built wheels are not available — if you encounter a `NotImplementedError`, ensure the build completed successfully.
 
 <details>
 <summary><strong>DGX Spark</strong> (tested with DGX Spark GB10)</summary>
@@ -240,7 +240,7 @@ See the [Orin setup guide](scripts/deployment/README.md#jetson-orin-setup) for D
 > destroy the platform-specific environment.
 
 
-For a containerized setup that avoids system-level dependency conflicts, see our [Docker Setup Guide](docker/README.md).
+For a containerized setup that avoids system-level dependency conflicts, see our [Docker Setup Guide](docker/README.md). The recommended container workflow is to start the image first, then clone or pull the repo inside the running container so your checkout uses the image's prebuilt dependency environment.
 
 ---
 
@@ -310,6 +310,8 @@ See the full [Data Preparation Guide](getting_started/data_preparation.md) for s
 
 ## Inference
 
+> **Prefer an interactive walkthrough?** The [`getting_started/GR00T_inference.ipynb`](getting_started/GR00T_inference.ipynb) notebook steps through loading the model and predicting actions from observations on a sample dataset.
+
 ### Zero-Shot Inference (Base Model)
 
 The included `demo_data/droid_sample` dataset works with the base model out of the box — no finetuning or checkpoint download needed:
@@ -326,8 +328,6 @@ uv run python scripts/deployment/standalone_inference_script.py \
 
 This runs open-loop inference on 2 DROID episodes, comparing predicted actions against ground truth. The base model downloads automatically from HuggingFace on first run (~6 GB).
 
-The standalone inference script defaults to the `ffmpeg` video backend so this demo works on systems with newer FFmpeg releases. If you explicitly use `--video-backend torchcodec`, make sure your installed `torchcodec` wheel is compatible with your system FFmpeg version.
-
 ### Finetuned Inference
 
 For posttrain embodiments, use a finetuned checkpoint. Most finetuned checkpoints (e.g., DROID, SimplerEnv) have a flat file structure and can be passed directly as a HuggingFace model ID — no manual download needed:

README.md

@@ -1,9 +1,10 @@
 # syntax=docker/dockerfile:1
 # GR00T Docker image (x86_64 + aarch64)
 #
-# Single Dockerfile for both architectures. On aarch64 (GB200, Grace Hopper)
-# flash-attn and torchcodec are installed from prebuilt wheels under
-# scripts/deployment/dgpu/wheels/ via pyproject.toml path sources.
+# Single Dockerfile for both architectures. On aarch64 (GB200, Grace Hopper),
+# flash-attn and torchcodec are installed from wheels under
+# scripts/deployment/dgpu/wheels/ via pyproject.toml path sources. If those
+# wheels are missing, the Docker build bootstraps them from source first.
 #
 # Build:
 #   docker build -f docker/Dockerfile -t gr00t .
@@ -17,6 +18,8 @@ SHELL ["/bin/bash", "-c"]
 
 ENV DEBIAN_FRONTEND=noninteractive \
     NVIDIA_DRIVER_CAPABILITIES=graphics,utility,compute \
+    GR00T_GLOBAL_VENV=/opt/gr00t-venv \
+    UV_PROJECT_ENVIRONMENT=/opt/gr00t-venv \
     PYTHON=/usr/bin/python \
     CUDA_HOME=/usr/local/cuda \
     PATH=/usr/local/cuda/bin:${PATH} \
@@ -68,7 +71,18 @@ apt_install_retry() {
             python3-pip \
             python-is-python3 \
             software-properties-common \
+            cmake \
+            ninja-build \
+            pkg-config \
+            pybind11-dev \
             ffmpeg \
+            libavdevice-dev \
+            libavfilter-dev \
+            libavformat-dev \
+            libavcodec-dev \
+            libavutil-dev \
+            libswresample-dev \
+            libswscale-dev \
             libegl1; then
             return 0
         fi
@@ -98,50 +112,57 @@ pip_install_retry
 curl --retry 5 --retry-delay 10 --retry-max-time 300 -LsSf https://astral.sh/uv/0.8.14/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
 INNEREOF
 
-WORKDIR /workspace
+WORKDIR /tmp/gr00t-venv-build
 
 # Install Python dependencies from lockfile
 # The wheels/ dir must be present before `uv sync` because pyproject.toml
 # references them as path sources for aarch64 flash-attn and torchcodec.
 COPY pyproject.toml uv.lock ./
+COPY scripts/deployment/dgpu/bootstrap_wheels.sh ./scripts/deployment/dgpu/bootstrap_wheels.sh
 COPY scripts/deployment/dgpu/wheels/ ./scripts/deployment/dgpu/wheels/
 
 RUN <<INNEREOF
 set -euo pipefail
 
-uv_sync_retry() {
+uv_lock_retry() {
     for attempt in 1 2 3 4 5; do
-        if UV_PREVIEW=1 UV_HTTP_TIMEOUT=300 UV_CONCURRENT_DOWNLOADS=4 uv sync --frozen --no-install-project --extra dev --no-cache; then
+        if UV_PREVIEW=1 UV_HTTP_TIMEOUT=300 UV_CONCURRENT_DOWNLOADS=4 uv lock; then
             return 0
         fi
         sleep $((attempt * 10))
     done
     return 1
 }
 
-uv_sync_retry
-INNEREOF
-
-# Install the gr00t project itself
-COPY . .
-RUN <<INNEREOF
-set -euo pipefail
-
-uv_pip_install_retry() {
+uv_sync_retry() {
     for attempt in 1 2 3 4 5; do
-        if uv pip install --python /workspace/.venv/bin/python -e . --no-deps; then
+        if UV_PREVIEW=1 UV_HTTP_TIMEOUT=300 UV_CONCURRENT_DOWNLOADS=4 uv sync --frozen --no-install-project --extra dev --no-cache; then
             return 0
         fi
         sleep $((attempt * 10))
     done
     return 1
 }
 
-uv_pip_install_retry
+if [ "$(uname -m)" = "aarch64" ]; then
+    bash scripts/deployment/dgpu/bootstrap_wheels.sh
+    uv_lock_retry
+fi
+
+uv_sync_retry
+sha256sum uv.lock | awk '{print $1}' > "${GR00T_GLOBAL_VENV}/.gr00t-uv-lock.sha256"
+mkdir -p /opt/gr00t-image-metadata
+cp pyproject.toml uv.lock /opt/gr00t-image-metadata/
+cd /
+rm -rf /tmp/gr00t-venv-build
 INNEREOF
 
-ENV PATH="/workspace/.venv/bin:${PATH}" \
-    VIRTUAL_ENV="/workspace/.venv"
+WORKDIR /
+RUN mkdir -p /workspace && ln -sfn "${GR00T_GLOBAL_VENV}" /workspace/.venv
+WORKDIR /workspace
+
+ENV PATH="${GR00T_GLOBAL_VENV}/bin:${PATH}" \
+    VIRTUAL_ENV="${GR00T_GLOBAL_VENV}"
 
 # EGL/Vulkan setup for headless rendering (MuJoCo, PyOpenGL)
 RUN mkdir -p /usr/share/glvnd/egl_vendor.d && \
@@ -169,7 +190,8 @@ ENV MUJOCO_GL="egl" \
     PYOPENGL_PLATFORM="egl" \
     __EGL_VENDOR_LIBRARY_FILENAMES="/usr/share/glvnd/egl_vendor.d/10_nvidia.json"
 
-# torchcodec and flash-attn for aarch64 are installed from prebuilt wheels
-# by `uv sync` above via pyproject.toml path sources — no source build needed.
+# torchcodec and flash-attn for aarch64 are installed from build-context wheels
+# by `uv sync` above via pyproject.toml path sources. Missing wheels are built
+# from source before sync, then committed back to the repo.
 
 CMD ["/bin/bash"]

docker/Dockerfile

@@ -18,23 +18,49 @@ From the repository root:
 bash docker/build.sh
 ```
 
-This builds from `nvidia/cuda:12.8.0-devel-ubuntu22.04`, installs all dependencies via `uv sync`, and sets up the GR00T codebase at `/workspace/`.
+This builds from `nvidia/cuda:12.8.0-devel-ubuntu22.04` and installs all dependencies into `/opt/gr00t-venv`. The image does not include a working source checkout; for normal use, start the image and then clone or pull the repo you want to run inside the container.
 
 ## Running the Container
 
-**Interactive shell (uses code baked into image):**
+**Recommended workflow: run the image, then clone or update the repo inside it.**
+
+Start an interactive shell:
+
 ```bash
 docker run -it --rm --gpus all \
     --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
     gr00t
 ```
 
-**Development mode (mounts local codebase for live editing):**
+Then, inside the container:
+
+```bash
+git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T /workspace/Isaac-GR00T
+cd /workspace/Isaac-GR00T
+export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
+python -c "import gr00t; print('GR00T ready')"
+```
+
+The image venv is active by default (`/opt/gr00t-venv`; `/workspace/.venv` is a compatibility symlink), and uv is configured with `UV_PROJECT_ENVIRONMENT=/opt/gr00t-venv`. After setting `PYTHONPATH` to the checked-out repo, both `python ...` and `uv run ...` use the global image venv instead of creating a checkout-local `.venv`. If you are working on an existing checkout in the container, run `git pull --ff-only` from that checkout instead of cloning again.
+
+The global venv records the `uv.lock` hash it was built from. If your checked-out repo uses a different lockfile, create a checkout-local venv before running commands. Reusing a uv cache keeps this path from starting cold:
+
+```bash
+export UV_CACHE_DIR="${UV_CACHE_DIR:-/workspace/uv-cache}"
+export UV_LINK_MODE=copy
+UV_PROJECT_ENVIRONMENT="$PWD/.venv" uv sync
+source .venv/bin/activate
+```
+
+Do not run a bare `uv sync` unless you intend to update the global image venv. Use `UV_PROJECT_ENVIRONMENT="$PWD/.venv" uv sync` when you want an isolated per-checkout environment.
+
+Avoid bind-mounting over `/workspace`, because that can hide the image's `/workspace/.venv` compatibility symlink. If you need to mount local source for live editing, mount it under a subdirectory:
+
 ```bash
 docker run -it --rm --gpus all \
     --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
-    -v $(pwd):/workspace \
-    gr00t bash -c "uv pip install -e . && bash"
+    -v "$(pwd):/workspace/Isaac-GR00T" \
+    gr00t bash -c 'cd /workspace/Isaac-GR00T && export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" && bash'
 ```
 
 ## Edge Device Containers

docker/README.md

@@ -99,7 +99,7 @@ NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=640 SAVE_STEPS=1000 uv run bash exa
 
 2. Install dependencies for the GR00T control script in the environment from step 1:
 ```bash
-pip install tyro moviepy==1.0.3 pydantic numpy==1.26.4
+pip install tyro pydantic numpy==1.26.4
 ```
 
 3. Enter the camera IDs for your ZED cameras in `examples/DROID/main_gr00t.py`.

examples/DROID/README.md

@@ -104,17 +104,16 @@ def _load_runtime_deps():
     try:
         import imageio
         from droid.robot_env import RobotEnv
-        from moviepy.editor import ImageSequenceClip
         from server_client import PolicyClient
         from utils import resize_with_pad
     except ModuleNotFoundError as exc:
         raise ModuleNotFoundError(
             "examples/DROID/main_gr00t.py requires the DROID robot-control environment. "
             "Follow examples/DROID/README.md and install the DROID package plus "
-            "`pip install tyro moviepy==1.0.3 pydantic numpy==1.26.4` before running."
+            "`pip install tyro pydantic numpy==1.26.4` before running."
         ) from exc
 
-    return RobotEnv, PolicyClient, ImageSequenceClip, imageio, resize_with_pad
+    return RobotEnv, PolicyClient, imageio, resize_with_pad
 
 
 # We are using Ctrl+C to optionally terminate rollouts early -- however, if we press Ctrl+C while the policy server is
@@ -140,7 +139,7 @@ def handler(signum, frame):
 
 
 def main(args: Args):
-    RobotEnv, PolicyClient, ImageSequenceClip, imageio, resize_with_pad = _load_runtime_deps()
+    RobotEnv, PolicyClient, imageio, resize_with_pad = _load_runtime_deps()
 
     assert args.external_camera in ["left", "right"], (
         f"Invalid exterior camera: {args.exterior_camera}"
@@ -380,9 +379,9 @@ def main(args: Args):
         save_filename = os.path.join(
             results_dir, "videos", f"{sanitized_instruction}_video_" + timestamp
         )
-        ImageSequenceClip(list(video), fps=args.render_fps).write_videofile(
-            save_filename + ".mp4", codec="libx264"
-        )
+        with imageio.get_writer(save_filename + ".mp4", fps=args.render_fps) as writer:
+            for frame in video:
+                writer.append_data(frame)
 
         if args.debug:
             model_wrist_image_writer.close()

examples/DROID/main_gr00t.py

@@ -282,6 +282,7 @@ def __init__(
         strict: bool = False,
     ):
         super().__init__(strict=strict)
+        self._closed = False
         self.context = zmq.Context()
         self.host = host
         self.port = port
@@ -335,10 +336,39 @@ def call_endpoint(
             raise RuntimeError(f"Server error: {response['error']}")
         return response
 
+    def close(self) -> None:
+        """Release the REQ socket and ZMQ context. Idempotent.
+
+        Mirrors ``gr00t.policy.server_client.PolicyClient.close()`` so the
+        DROID example demonstrates the explicit-lifecycle pattern.
+        """
+        if getattr(self, "_closed", True):
+            return
+        self._closed = True
+        socket = getattr(self, "socket", None)
+        if socket is not None:
+            try:
+                socket.close(linger=0)
+            except Exception:
+                pass
+        context = getattr(self, "context", None)
+        if context is not None:
+            try:
+                context.term()
+            except Exception:
+                pass
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc, tb):
+        self.close()
+
     def __del__(self):
-        """Cleanup resources on destruction"""
-        self.socket.close()
-        self.context.term()
+        try:
+            self.close()
+        except Exception:
+            pass
 
     def _get_action(
         self, observation: dict[str, Any], options: dict[str, Any] | None = None

examples/DROID/server_client.py

@@ -36,12 +36,12 @@ Provided checkpoints:
 | Task | N1.6 success rate | N1.7 success rate |
 | --- | ---: | ---: |
 | `google_robot_pick_coke_can` | 95/100 (95.0%) | 100/100 (100.0%) |
-| `google_robot_pick_object` | 87/100 (87.0%) | 96/100 (96.0%) |
-| `google_robot_move_near` | 81/100 (81.0%) | 96/100 (96.0%) |
-| `google_robot_open_drawer` | 0/100 (0.0%) | 36/100 (36.0%) |
-| `google_robot_close_drawer` | 44/100 (44.0%) | 63/100 (63.0%) |
-| `google_robot_place_in_closed_drawer` | 5/100 (5.0%) | 21/100 (21.0%) |
-| **Average** | **52.0%** | **68.7%** |
+| `google_robot_pick_object` | 87/100 (87.0%) | 94/100 (94.0%) |
+| `google_robot_move_near` | 81/100 (81.0%) | 100/100 (100.0%) |
+| `google_robot_open_drawer` | 0/100 (0.0%) | 65/100 (65.0%) |
+| `google_robot_close_drawer` | 44/100 (44.0%) | 69/100 (69.0%) |
+| `google_robot_place_in_closed_drawer` | 5/100 (5.0%) | 7/100 (7.0%) |
+| **Average** | **52.0%** | **72.5%** |
 
 # Fine-tune Simpler Env bridge dataset (WidowX robot)
 
@@ -74,7 +74,7 @@ uv run hf download \
 
 # Copy the patches and run the finetune script
 cp -r examples/SimplerEnv/fractal_modality.json examples/SimplerEnv/fractal20220817_data_lerobot/meta/modality.json
-uv run python examples/SimplerEnv/convert_av1_to_h264.py examples/SimplerEnv/fractal20220817_data_lerobot --jobs 16  # (Optional) if AV1 doesn't work on your machine
+uv run python examples/SimplerEnv/convert_av1_to_h264.py --root examples/SimplerEnv/fractal20220817_data_lerobot --jobs 16
 ```
 
 ```bash