Simple Actors and Deep Critics for Scalable Reinforcement Learning
Guhyeon Kang, Jaehwi Lee, Minhae Kwon
cs.LG
2026-08-27
LAC puts capacity in a 32-layer critic discarded at deploy; a tiny deterministic actor hits 70% mean OGBench success vs 36% for the strongest flow baseline, at 0.24ms per action.
Offline RL has spent the last two years putting policy expressivity into diffusion and flow-matching actors. They fit multimodal action distributions in logged data. The cost is paid at every decision: multiple denoising or integration steps per action. On robots and embedded controllers, that bill compounds.
Actor and critic are not symmetric across the train-deploy cut. The critic shapes the policy gradient and is thrown away after training. The actor runs on every step in production. Capacity in the critic is a one-time training cost. Capacity in the actor is a recurring inference cost. Deepening an MLP critic in offline RL has historically collapsed training, which is why published critics stayed two or three layers deep.
LAC (Light Actor, deep Critic) treats that asymmetry as the design. The actor is a small deterministic MLP, trained like TD3+BC: maximize the critic score plus a behavior-cloning penalty that keeps actions near the dataset. One forward pass at deploy time.
Deep critics fail in three independent ways, and LAC puts one fix on each:
n-step barely helps a shallow critic, because the noise loop is still dormant. The categorical loss is not interchangeable with MSE even when n-step is in place. Default critic: 32-layer residual MLP, about 2.15M parameters, 51 categorical atoms, n=4. Two actors: LAC-S is [256, 256] at about 0.13M parameters; LAC-L is [512]×4, sized to match the flow baselines.
Evaluation is OGBench, 7 environments × 5 task variants, 4 seeds, success rate. Latency is wall-clock per action on one RTX 3090 at batch size 1.
| Method | Mean success (7 envs) | Latency / action |
| BC | 3% | not reported |
| ReBRAC | 30% | 0.27ms |
| FQL (distilled flow) | 36% | 0.27ms |
| LAC-S | 52% | 0.24ms |
| LAC-L | 70% | 0.31ms |
| IDQL (10-step diffusion) | 15% | 0.95ms |
The gap is largest on long-horizon locomotion: on hum-large, LAC-L reaches 69±15 against IFQL at 11±2; on puzzle-3x3, LAC-L hits 95±1 against FQL at 30±1. On scene, LAC does not win: FQL is 56±2, LAC-L 41±39 with high variance.
Ablations line up with the three failure modes. On the task1 variants, the full recipe averages 73. Drop n-step and it falls to 33. Swap categorical for MSE and it falls to 37, with hum-large collapsing from 79 to 7 mid-training. Drop both and the average is 9. In the depth × n heatmap, nothing works below 8 layers. Past 16 layers, only n=1 degrades with depth (42% at 32 layers, 14% at 256); n≥2 stays stable, and n=4 is strongest. n=8 falls back to the level of n=2, so longer horizons are not automatically better.
Dropping the same critic into other actors still helps. On hum-medium task1, TD3+BC goes from 10 to 46 (+36), FQL from 19 to 57 (+38), and distilled-diffusion CAC from 38 to 97 (+59). Versus LAC-S latency, IDQL is 3.96× slower and IFQL 2.67× slower. Against already-distilled one-step methods (FQL, SRPO, CAC), the latency edge is tiny.
A clear deploy-time tradeoff: multimodal actions do not have to live in a generative actor. If the critic is strong enough, a small deterministic policy can amortize the signal into one forward pass. For robots and edge controllers that count milliseconds per step, this is more direct than distilling another diffusion policy.
The critic recipe also works as a drop-in. Keep the actor family, swap the critic, and five methods in the paper all go up. This is an incremental recipe, not a new algorithm class. It does make the "where to spend capacity" question concrete. The deterministic actor is the cheapest point on this spectrum, not proof that expressivity is redundant: CAC plus the same critic reaches 97%, so generative actors still have headroom on high-dimensional actions.
The authors restrict evaluation to state-based OGBench: no pixels, no real robots. LAC loses on scene, and the variance is high, so the expressivity gap of a deterministic actor is still real on sequential manipulation with sparse rewards. n has to be tuned; n=8 regresses. Most baseline numbers are copied from the FQL paper; only TD3+BC was rerun, so cross-implementation gaps remain. The near-4× latency claim is relative to multi-step generative actors, not to already-distilled one-step policies.