stable-envs speeds LeWM training up to 1.33× on four GPU-rendered sims

stable-envs-v1: Fast and Reproducible Environments with Controllable Variations

Sami BuGhanem, Francesco Capuano, Ahitagni Das, Richard Gao, Randall Balestriero

cs.SE

2026-11-18

stable-envs renders 80 sims on GPU, pure rendering hits 24.1× the reference, and four-env LeWM training hits 1.16× to 1.33× with planning unchanged.

What problem this solves

Pixel world models learn dynamics from images. Online interaction and offline collection both end at the same bottleneck: high-dimensional frames have to enter memory and then move onto the compute device. LeWM, a compact image-only model, already implies about 200GB of images on a medium task-specific set. The footnote's arithmetic is uint8, one byte per pixel, 224×224, roughly 20,000 trajectories of 200 steps. A maze is already awkward to store. A high-fidelity robot simulator is worse.

Compressing observations as video saves disk and spends the savings on decode. Loading, decoding, and staging frames for a large set can take hours. Streaming that decode next to the optimizer skips the prep step, then the GPU sits idle if frames arrive at the wrong rate. Either way, decoded images usually still cross from CPU to GPU.

Simulators can render on demand. Doing it on the GPU drops the stored frames and the host-to-device copy, and rendering throughput becomes the new limit. CPU vectorization and process pools increase how many environments you can run. They do not promise a faster draw for any one of them. Isaac Lab, ManiSkill3, and MuJoCo Playground already get high throughput from GPU physics and rendering, tied to specialized backends. Isaac Lab's GPU renderer, specifically, does not support Hopper-class GPUs such as the H100, and it does not support macOS.

stable-envs, from Brown, Oxford, and Rice, is a small library aimed at that gap. It batch-renders 80 environments on the GPU, restores full state exactly, and changes appearance while the physical state stays put.

Method

Physics for each environment is ported or rewritten, and the internal state is exposed in full. Draw speed, exact checkpoints, and separate control of appearance all sit on that state, behind one API. The 80 environments include TwoRoom and PushT, and newer ones such as RoadSter. Each class keeps its own low-level code. The library adds factors of variation such as color, batched calls, and checks against the source. Every instance goes through one scheduler, the simulation adapter: vectorized code or a process pool on CPU, and a batched physics backend on GPU, MuJoCo Warp or MJX, or a PyTorch kernel, chosen per environment.

One call runs B environments. reset returns a batched state. step accepts a single action [B, A] or a chunk [B, T, A], and rendering can be turned off so a physics-only rollout does not draw frames. render then draws that state to RGB at any resolution. The state stores both deterministic and stochastic variables, which is why a recorded state can be replayed. A dataset can keep those states instead of images and render at train time, at another resolution or with another color. That works because the simulator treats state and pixels as a one-to-one pair.

Two choices produce the speed. Redrawing static scenery every step wastes work, so each configuration is painted once per batch and later frames add only the moving bodies. On GPU, MuJoCo Warp and nvdiffrast, NVIDIA's rasterizer, write tensors on the training device and skip the trip back to the host. Appearance is a runtime call, setvariation. In PushT the T block can be set to RGB (230, 80, 60). Physics stays, pixels change. Visual factors can be sampled on their own.

Results

The reported ratios measure different things.

Pure rendering of prerecorded states, in parallel, to fill a training batch, runs at 1.4× to 24.1× the reference implementation on GPU. Against CPU-native baselines, environment throughput reaches 8.5×. Figure 1 attaches 8.5× to rendering. The conclusion's ceiling of about 24× is the top of that 24.1× range. Change the environment, or switch from pure rendering to a full training run, and the ratio drops by about an order of magnitude.

The end-to-end workload is LeWM, trained through stable-worldmodel, on only TwoRoom, PushT, Reacher, and Cube. The training loop is untouched except that frames now come from the GPU. Speed is 1.16× to 1.33× the reference, GPU utilization is higher, and downstream planning is described as comparable. The abstract says performance is not lost. Neither the utilization points nor the planning gap appear as numbers in the prose.

Dynamics parity uses one initial state and actions a0 through a20. Maximum error is in Table 7 and is not quoted. Images are compared by pixel MSE. On CPU the four environments are nearly pixel-identical. On GPU, TwoRoom, PushT, and Cube are almost exact. Reacher differs by 2.2, which is called negligible.

MeasurementBaselineResult
Batched pure rendering of stored statesReference, GPU1.4× to 24.1×
Environment throughputCPU-nativeup to 8.5×
End-to-end LeWM trainingReference, four envs1.16× to 1.33×
GPU pixel MSESource renderer2.2 on Reacher, others nearly exact
Downstream planningOriginal training runcalled comparable, no numeric gap

Why it matters

The algorithm does not change. The simulator stack does, by a modest amount. People training on pixels inside a simulator can plug it in.

A 200GB frame dump can become stored state, rendered when a batch is needed. Video decode and the CPU-to-GPU image copy go away. The same states can be redrawn at a new resolution or color without a new collection pass. All 80 environments share the interface.

The time saved is in drawing. The optimizer section of the run is the same length. Render throughput tops out at 8.5× a CPU baseline, and pure rendering reaches 24.1× the reference, while training speed rises only 16% to 33%. Model updates still dominate. Runs get shorter. They do not get cut in half.

Limitations

There is no throughput comparison with Isaac Lab, ManiSkill3, or MuJoCo Playground. Those three appear only as examples of backends and hardware that are locked together. Whether the GPU path runs on an H100 or on macOS is not measured. Drawing also depends on nvdiffrast. Portability is a motive. The measurements have not caught up.

The detailed speed and correctness results cover 4 environments, while the library claims 80. Dynamics are checked for about 21 steps, so drift on long rollouts is unknown. No loss of performance has no planning score behind it. The Reacher figure of 2.2 does not say whether it is a mean or a sum, or whether pixels are on a 0–255 scale.

Appearance control has no training evidence either. The abstract ties on-the-fly visual changes to more diverse data. No ablation tests that claim. What is actually shown is faster drawing, and short trajectories whose motion and pixels mostly match the source on four environments.

Terms

Source

What people are saying

Related papers

All paper explainers