MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes
Remco Hendriks
Continker
cs.LG, cs.AI, cs.CL
2026-09-09
A PEFT-tuned 4B Qwen scores 91.3 Tier 1 on 238 held-out MetroLLM-Bench cases, matching GPT-5.4; 9B and 27B students add nothing, while a rule agent hits 84.6.
Transit kiosks encode fare rules, network topology, and disruption responses as programmed state machines. Reflecting a station closure, a new fare bracket, or a holiday schedule typically means a developer ticket, an integrator patch, regression tests, and a release window. Rules change often, and a wrong fare is a billing error. That code cycle has a concrete cost.
MetroLLM-Bench tests a different runtime. The policy layer is a language model. It reads a natural-language system description (a framebook: terminology, currency, hours, cultural conventions), calls structured tools inside a ReAct loop of at most 20 rounds, and submits a machine-renderable terminal state: an outcome, a per-ticket fare quote when applicable, advisory banners, and a kiosk action. Display slots are fixed. The hardware has to act on the result. Operators still need to explain what the kiosk did.
Six real networks stretch the dimensions a policy layer must absorb: MARTA in Atlanta (38 stations), Doha Metro (37), BART (50), Taipei MRT (107), CTA (142), and Beijing Subway (414). Three fare models, four currencies. 955 cases sit in eleven categories: routing, fare, disruption, accessibility, cultural, policy, multi-turn, adversarial, temporal, tool-hallucination, and compound stress. A system-stratified 75/25 split, frozen before any training, puts 717 cases in the teacher-trace pool and 238 in held-out evaluation.
The six tools are routeplanner, farecalculator, stationinfo, lineinfo, disruptionfeed, and knowledgebase. Every submission must land on one of five outcomes. Pydantic validates the structure; an inconsistent state comes back as HTTP 422 so the model can repair it within the remaining round budget. Prompts and tool results supply the operational facts a case needs. The test is whether the model follows the supplied rules, not whether it memorised a network in pretraining.
Scoring has two tiers. Tier 1 holds fourteen deterministic components: route and fare correctness, no tool hallucination, renderable state, outcome and reason codes, fare breakdown, purchase gate, disruption detection, re-planning efficiency, and related checks. That tier is clean enough to serve as the PEFT reward. Tier 2 holds eight semantic-quality components, six of them judged by Claude Haiku 4.5. Reported numbers are Tier 1 and the composite of both tiers.
The PEFT recipe is narrow. Teachers are Qwen 3.5 27B-dense and 35B-A3B, run only on the training partition. Traces with Tier 1 of at least 90 percent are kept, then deduplicated by case to 600 examples with a mean of 99.0. Students at 2B, 4B, 9B, and 27B share QLoRA rank 16, three epochs, and an effective batch of 8. The 27B run drops max sequence length from 4096 to 2048 to fit 32 GB of VRAM. Nine training runs took 9.4 GPU-hours on one RTX 5090.
On the 238 held-out cases, Muse Glimmer 30B leads the composite at 92.03, with Tier 1 at 92.75. Qwen3.6-27B has the highest Tier 1, 93.63. The row that matters for deployment is the 4B PEFT student: 2.6 GB Q4KM, Tier 1 91.32, above GPT-5.6 luna (90.63) and sol (90.00), and 0.05 below GPT-5.4 full at xhigh effort (91.37), inside the paired-bootstrap interval.
| Model | held-out Tier 1 | composite |
| Muse Glimmer 30B | 92.75 | 92.03 |
| Qwen3.6-27B | 93.63 | 91.28 |
| GPT-5.4 full (xhigh) | 91.37 | 90.45 |
| Qwen 3.5 4B + PEFT | 91.32 | 89.12 |
| GPT-5.6 luna | 90.63 | 90.57 |
| Rule-based baseline | 84.6 | 77.1 |
PEFT gain over the matching base falls as size rises: +7.03 at 2B (three seeds), +2.00 at 4B, +1.65 at 9B, −0.91 at 27B. Every seed agrees on the sign at every size. The 4B, 9B, and 27B students sit between 91.03 and 91.41 on held-out Tier 1, a 0.38-point band. A scripted agent that chains tools in a fixed order reaches 84.6 Tier 1 and 77.1 composite. It is already close on routing (93.5 composite) and fare (91.7). Temporal drops to 52.1, accessibility to 69.7, policy to 73.3. The language-model margin lives in the categories that require a decision.
Serving configuration moves the board. Under uniform greedy decoding and a 4096-token output budget, Qwen3.8-27B trails Qwen3.5 by 3.60 Tier 1 points. Raising the budget and switching to vendor-recommended sampling accounts for about 2.7 of that gap. The leaderboard therefore ranks each of those rows at its own best config. Selection optimism is unquantified.
Judge calibration is only moderate. Quadratic-weighted κ is 0.53 between the author and Haiku, 0.25 between the two human raters. Headline comparisons use deterministic Tier 1.
For anyone shipping a narrow tool-using agent on local hardware, this is a reproducible existence proof. Bound the task tightly enough and a 2.6 GB open-weight PEFT student can match a frontier API on the executable contract. Weights, adapter, prompts, tools, and inference can stay inside the operator's stack.
The PEFT boundary is equally clear. Once the base is already competent, the same 600 traces, rank-16, three-epoch recipe stops helping. On the full 955-case matrix, 27B adaptation significantly reduces Tier 1. The unadapted 27B base remains the strongest of the four sizes. Smaller models are not intrinsically better. Extra capacity and an adapter need not help a model that already does the job.
Cases, harness, rule agent, and the four students are public.
The task is deliberately bounded: six tools, typically three to seven calls, a constrained terminal state. On Temporal, GPT-5.4 xhigh still leads by 13.7 composite points (87.2 against 73.5 for Qwen 27B base). Frontier rows also keep an edge on adversarial cases. The 4B student trails GPT-5.6 luna and GPT-5.4 on composite. The numbers do not support exporting this parity to broader action spaces or longer horizons.
The 238-case held-out set is underpowered. All four primary paired intervals include zero; the 4B gain and 27B regression become significant only on the full matrix. The top eleven models sit within 3.18 composite points, comparable to single-run noise, so the top of the board is weakly ordered. Most non-PEFT models have one run. OpenAI rows are stuck at temperature 1.0.
The 27B student trained at 2048 tokens, so the negative delta may mix in that hardware constraint. Teachers and students are the same family. Cases have been public since 19 June 2026; Muse Glimmer and Qwen3.8 shipped after that date, so contamination cannot be ruled out from the public record. All answers are in English, including for non-English systems. The eval does not cover payments, assistive hardware, PII, or end-to-end latency. One author wrote the framebooks and cases with AI assistance, and a few known answer-key defects were left unfixed so the evaluation set would not move mid-campaign.