Skip to content
Closed
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
307543a
Started prototyping. Got a recolor event working in the notebook. Nee…
alexmillane Apr 21, 2026
9e5c71c
Prototype of the variations interface.
alexmillane Apr 22, 2026
37ec8da
Add variations to objects.
alexmillane Apr 22, 2026
3d27adb
Variations as concrete classes.
alexmillane Apr 22, 2026
fb5a6f2
Fixed recursive descent.
alexmillane Apr 23, 2026
f519018
Different colors per object in randomize_visual_color.
alexmillane Apr 23, 2026
bab514f
Update the plan and status.
alexmillane Apr 23, 2026
e618564
Move to cfg based variataion configuration.
alexmillane Apr 24, 2026
a36ffa7
Variations schema.
alexmillane May 11, 2026
e638d1f
Add overriding cfg struct via string options.
alexmillane May 11, 2026
3f9dc55
Got policy_runner working with the hydra overrides.
alexmillane May 12, 2026
f0bdde9
Experimentation with event wrappers.
alexmillane May 20, 2026
36280ba
Add a version of the color event that uses a sampler.
alexmillane May 20, 2026
3165b7a
Update sampler to move ledger writing to the base class.
alexmillane May 20, 2026
ef25425
Move the ledger registration to the snv builder and output in example…
alexmillane May 20, 2026
34b4817
Agentic cleanup of the docstrings and asserts.
alexmillane May 21, 2026
16cb604
Move variations hydra out of the compiler
alexmillane May 21, 2026
3bd6fdc
Rename ledger to variation_recorder.
alexmillane May 21, 2026
1dd82d5
remove variation registry
alexmillane May 21, 2026
68abea0
Self review.
alexmillane May 21, 2026
fe45e7e
More cleanup
alexmillane May 21, 2026
c4e96a2
Build time variations. Not fully functional yet.
alexmillane May 26, 2026
0ff7de9
Add extrinsics variation, not yet wired up in the compiler.
alexmillane May 26, 2026
1d3c17d
Source variations also from embodiment, as well as scene.
alexmillane May 26, 2026
b3e7086
Upgrade compile env notebook to test the camera extrinsics variation
alexmillane May 26, 2026
7322dcd
Fix issue with variations recorder.
alexmillane May 28, 2026
ee97249
Messing around with OpenGL transforms until they worked.
alexmillane May 28, 2026
e700ceb
Correct extrinsic decalibration in the variation term
alexmillane May 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
611 changes: 611 additions & 0 deletions 2026_04_13_sensitivity_analysis.md

Large diffs are not rendered by default.

23 changes: 23 additions & 0 deletions 2026_04_21_color_variation_status.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Color Variation — POC Status

Companion to [2026_04_21_variation_system_plan.md](2026_04_21_variation_system_plan.md).

## Status: shipped as `ObjectColorVariation`

Per-env flat-colour randomisation works end-to-end on any `Object`, via the variation system. Validated in `isaaclab_arena/examples/compile_env_notebook.py` with `num_envs=4`, the kitchen background, and two YCB objects (`cracker_box`, `tomato_soup_can`):

```python
cracker_box.get_variation("color").enable()
tomato_soup_can.get_variation("color").enable()
```

Each cloned env gets a distinct random flat colour bound to the object's top-level prim. Requires `scene.replicate_physics=False` (Arena default).

## Known limitations

- **Texture is dropped.** The event goes through `mdp.randomize_visual_color`, which replaces the bound material with a fresh `OmniPBR` whose `diffuse_color_constant` is randomised. The original diffuse texture is lost. An in-place tint path (preserving the texture) was prototyped in `isaaclab_arena/examples/tint_events.py` / `randomize_visual_diffuse_tint` — runs without error but the render doesn't change. Left in the notebook for A/B when we come back to it.
- Only `UniformSampler` over RGB is supported. Discrete palettes (`randomize_visual_color`'s list-of-tuples path) need a `DiscreteChoiceSampler`.

## Next

- Explore how to do configurations of the variations from the command line using Hydra.
109 changes: 109 additions & 0 deletions 2026_04_21_variation_system_plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Variation System — Plan

Companion to [2026_04_13_sensitivity_analysis.md](2026_04_13_sensitivity_analysis.md). First slice of the sensitivity-analysis feature: the *variation system* only. Analysis tooling is deferred.

## Goal

Build a variation system (samplers + variation base + registry) and validate it end-to-end with one concrete variation: `ObjectColorVariation` using a `UniformSampler` over RGB. No CLI, no eval_runner config, no other variations yet. Status: shipped — see [2026_04_21_color_variation_status.md](2026_04_21_color_variation_status.md).

## Design

Every `Object` subclass attaches the variations it supports in `__init__`, disabled by default. Users opt in via `get_variation(name).enable()` and optionally narrow the distribution with `set_sampler(...)`. The builder walks the scene, collects enabled variations, and merges their event terms into `events_cfg`.

```mermaid
flowchart LR
Ctor["Object.__init__\nself.add_variation(ObjectColorVariation(self))"] --> Inst[Object instance\n_variations dict]
User["cracker_box.get_variation('color').enable()"] --> Inst
Inst --> Scene["Scene.get_variations()"]
Scene --> Builder["ArenaEnvBuilder._compose_variations_event_cfg"]
Builder -->|"variation.build_event_cfg(scene)"| ETC[EventTermCfg]
ETC --> Events[events_cfg]
```

Key separations:

- **Sampler**: stateless distribution (`Sampler` ABC + `UniformSampler`). RNG passed in at sample time.
- **Variation**: one knob. Owns a sampler, remembers its target asset *by name only* (see "gotchas"), emits an `EventTermCfg`.
- **Asset**: instantiates its supported variations in `__init__`, disabled. User flips them on.
- **Registry**: global `name → Variation class` table populated by `@register_variation`. Naming contract for later CLI resolution; not consumed by the builder yet.

## Module layout

- `isaaclab_arena/variations/__init__.py` — public re-exports; import triggers registrations.
- `isaaclab_arena/variations/sampler.py` — `Sampler` ABC + `UniformSampler`.
- `isaaclab_arena/variations/variation_base.py` — `VariationBase` ABC.
- `isaaclab_arena/variations/variation_registry.py` — `VariationRegistry` + `@register_variation`.
- `isaaclab_arena/variations/object_color.py` — `ObjectColorVariation` (first concrete variation).

## Core interfaces

```python
class Sampler(ABC):
def sample(self, num_samples, generator=None) -> torch.Tensor: ...

class UniformSampler(Sampler):
def __init__(self, low, high): ... # scalar or broadcastable sequence

class VariationBase(ABC):
name: ClassVar[str]
def enable(self): ...
def set_sampler(self, sampler: Sampler): ...
@abstractmethod
def build_event_cfg(self, scene: Scene) -> tuple[str, EventTermCfg]: ...

@register_variation
class ObjectColorVariation(VariationBase):
name = "color"
def __init__(self, asset: ObjectBase, mode="reset", mesh_name=""):
self.asset_name = asset.name # name only, no back-ref to the asset
```

Attached from the asset's `__init__`:

```python
class Object(ObjectBase):
def __init__(self, ...):
...
self.add_variation(ObjectColorVariation(self)) # disabled until .enable()
```

## Gotchas we hit

- **Don't back-reference the asset on the variation.** `configclass._validate` walks `obj.__dict__` recursively with no cycle check; `Object._variations["color"].asset → Object` creates an unbounded reference cycle that explodes validation with `RecursionError`. `VariationBase` stores nothing about the asset; concrete subclasses store the asset *name*.
- **`randomize_visual_color` RNGs collide.** Upstream seeds its `ReplicatorRNG` from the global seed with no per-term entropy, so two enabled colour variations emit byte-identical colour streams. `ObjectColorVariation` wires a local `_PerEventSeededRandomizeVisualColor` subclass that re-seeds the RNG from a hash of the (unique) `event_name`.
- **`scene.replicate_physics` must be `False`** for per-env material divergence. Newton preset flips it on; the example notebook asserts against it at compose time.

## Integration touchpoints

- `isaaclab_arena/assets/object_base.py` — `add_variation`, `get_variation`, `get_variations`, `_variations` dict.
- `isaaclab_arena/assets/object.py` — `self.add_variation(ObjectColorVariation(self))` in `__init__`.
- `isaaclab_arena/scene/scene.py` — `Scene.get_variations()` walks `ObjectBase` assets.
- `isaaclab_arena/environments/arena_env_builder.py` — `_compose_variations_event_cfg()` merges enabled variations into `events_cfg` (asserts unique event names).

## Example

```python
cracker_box = asset_registry.get_asset_by_name("cracker_box")()
cracker_box.get_variation("color").enable()
# optional: cracker_box.get_variation("color").set_sampler(
# UniformSampler(low=(0.4,)*3, high=(1.0,)*3)
# )
scene = Scene(assets=[cracker_box, ...])
```

See `isaaclab_arena/examples/compile_env_notebook.py`.

## Open todos

- [ ] `isaaclab_arena/tests/test_variations.py` — sampler/registry unit tests + a ≥2-env sim integration test asserting per-env colour divergence.
- [ ] `ObjectMassVariation` — exercise the abstraction on a non-visual knob and a non-Replicator event path.
- [ ] `DiscreteChoiceSampler` — palette-style sampling; covers `randomize_visual_color`'s list-of-tuples path.
- [ ] Env-level variations escape hatch (scene-wide lights, HDR) — `IsaacLabArenaEnvironment.variations` + `add_variation` helper.
- [ ] CLI / Hydra plumbing for variation selection (`--variation cracker_box.color=uniform(...)`).

## Out of scope (this slice)

- Analysis tooling / sensitivity metrics.
- Per-env sampled-value logging for downstream analysis.
- Registration-time vs run-time orchestration (POC is per-reset runtime only).
- Semantic target indirection (`"pick_up_object"` → asset).
29 changes: 29 additions & 0 deletions 2026_04_21_variations_user_interface.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

## Variations in user interface in Python

# Option 1: Variations travel with asset.
# DECISION: SUPPORTED
# Reason: Certian variations will be specific to a single asset, for example
# embodiment specific variations. Therefore it makes sense that variations
# travel with the asset. The second thing is that then, for default variations
# the user doesn't need to deal with them in the environment file, they are
# automatically there (disabled by default).

asset_registry = AssetRegistry()
apple = asset_registry.get_asset_by_name("apple")
apple.get_variation("color").enable()
apple.get_variation("color").set_sampler(UniformSampler(low=(0.0,) * 3, high=(1.0,) * 3))

# Option 2: Variations are added objects.
# DECISION: AGAINST

asset_registry = AssetRegistry()
apple = asset_registry.get_asset_by_name("apple")
color_variation = ObjectColorVariation(apple, sampler=UniformSampler(low=(0.0,) * 3, high=(1.0,) * 3))


## Variations in user interface in Hydra
Loading