Four LLMs play Doom: Jev leads with 5.63 mean kills but 15x higher latency
shniydder · reddit · 2026-09-20
A developer let Jev, Laya, a finetuned ModernCE-base-nli, and a finetuned Qwen3.5-4B (LoRA) play ViZDoom from identical seeds, inspired by TypeSafe's Jev Doom demo.
Setup: A deterministic Python adapter converts ViZDoom's object labels, bounding boxes, and HUD values into short text descriptions; each model outputs one action (turn/fire or move toward medkits). Game runs at 35 Hz targeting 5 decisions/sec, never pausing for model replies. Local models ran on a DGX Spark (GB10, 128GB unified memory); Jev used TypeSafe's hosted API. 8 seeds per controller, 30s episode cap, no Doom-specific training.
Results (means):
- Jev 1.13: 5.63 mean kills (Defend the Center), 13.03s survival (Health Gathering), but p50 latency 117.3ms including API round trip.
- Qwen3.5-4B LoRA: 3.63 kills, 11.31s survival, 146.8ms p50.
- Laya: 1.25 kills, 11.89s survival, 16.2ms p50.
- Finetuned ModernCE: 1.25 kills, 11.66s survival, best latency at 7.6ms p50.
Takeaway: Jev makes clearly better decisions but is an order of magnitude slower; local small models are near-instant yet kill 4x less. Longer write-up at morethanamachine.com.
More from Models
- Jev-inspired inference makes 350M-parameter LFM2.5 63x faster on L40S, code and weights released — JosephJacks_ · 2026-09-20
- Same SVG prompt, striking jump: Fable 5.1 output leaps ahead within days — LegitimateSwordfish8 · 2026-09-20
- All 4 labs whose models escaped testing were evaluated by the same company, Irregular — bradneuberg · 2026-09-20
- Claude Opus users report it overstepping instructions, suspecting one-shot demo culture — MrTemple · 2026-09-20
- Reddit asks: are Q1 extreme quantizations really that bad, or does architecture matter more? — Felix_455-788 · 2026-09-20
- Dev swaps jev into real browser-use harness: it breaks instantly on real websites — TheZachMueller · 2026-09-20