FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth
Josef Chen, Erim Hayretci
cs.AI, cs.CY, cs.LG, cs.SE
2026-08-21
FlavourBench scores three-of-eight portfolios with frozen Epicure utilities, no model judge. Grok 4.6 leads at 65.1 on 534 shared tasks; 101 of 351 pairs resolve after Holm.
Open-ended language-model benchmarks almost always inherit a judge: a human preference panel, another model, or a brittle exact-match key. The judge gets entangled with the systems under test. Exact match then collapses a decent answer and a terrible one into the same miss. Code can run tests. Databases can be inspected. Games can be scored. Culinary reasoning usually cannot.
FlavourBench turns a versioned culinary runtime into an executable environment. Epicure represents 1,790 ingredients in 300 dimensions and exposes deterministic operations for substitution, pairing, dietary feasibility, and regional composition. It is not a claim of perfect human taste, and it is not a leaderboard entry. It defines the task reward surface. The score measures agreement with that published environment.
Each task shows eight candidate ingredients and asks for a three-item portfolio, 56 combinations. Before any model runs, Epicure assigns a raw utility to every portfolio and min-max normalizes inside the task to 0–100. The unique optimum is always 100. Invalid portfolios under a diet or category constraint score zero.
The ranked core has 534 tasks from two independently compiled panels, 178 per family:
| Family | Decision | Utility mix | Chance | Median top gap | Distinct scores |
| Substitution | three-item replacement for an anchor | 0.8 similarity + 0.2 coherence | 45.2 | 5.6 | 56 |
| Pairing | three accompaniments for an anchor | 0.65 affinity + 0.35 coherence | 45.0 | 5.6 | 56 |
| Constraints | feasible set under diet and NOVA caps | 0.7 similarity + 0.3 coherence | 3.4 | 37.3 | 4 |
Constraint tasks are sparse on purpose: infeasible means zero. Regional-composition tasks ship as a supplemental track and stay off the 27-model board, because they cannot support an equally complete common core.
The last prompt line must start with FINALSELECTION and three A–H labels. The parser reads only that line and does one lookup. Prose does not change the score. Epicure is not exposed as a tool during ranking, so the metric is whether the model can make the culinary decision, not whether it can echo a runtime return value.
Twenty-seven endpoints span OpenAI, Anthropic, Google, xAI, through Cohere. Every ranked model contributes exactly 89 valid responses per panel and family, 14,418 cells. A task enters the common core on response status and parser validity only; task IDs are frozen before scores are loaded. Endpoints with a reasoning-effort knob have hidden chain-of-thought excluded and effort pinned to the lowest transport-stable setting.
The FlavourBench Score is the equal mean of six panel-family strata. Inference uses 50,000 anchor-cluster bootstrap draws for simultaneous 95% bands and 100,000 sign-flip tests on all 351 pairs, Holm-controlled. Tasks that share an ingredient anchor move together.
Grok 4.6 has the largest point estimate, 65.1, simultaneous 95% interval [61.0, 69.2], bootstrap rank interval [1, 5]. The top of the table sits in one unresolved group. Adjacent ranks are not a settled order.
| Rank | Model | Score | Simultaneous 95% CI | Group |
| 1 | Grok 4.6 | 65.1 | [61.0, 69.2] | 1 |
| 2 | Gemini 3.1 Pro | 65.0 | [60.8, 69.1] | 1 |
| 3 | GPT-5.6 Sol Pro | 64.2 | [60.1, 68.4] | 1 |
| 26 | Llama 4 Maverick | 53.7 | [49.6, 57.7] | 3 |
| 27 | Command R+ | 47.9 | [43.7, 52.0] | 3 |
After Holm, 101 of 351 pairs remain significant. Group 1 runs from Grok 4.6 down through GLM 5.3 at 60.6. Group 3 is only Llama 4 Maverick and Command R+. Panel-level scores correlate at Pearson r=0.89 and rank ρ=0.80.
Family scores split models that look similar in aggregate. Kimi K3 posts the best pairing score at 73.7 and only 47.0 on constraints. Gemini 3.1 Pro is 69.0 / 71.6 / 54.2 on substitution, pairing, and constraints. Constraints, which zero out illegal portfolios, separate the field the most. Chance is about 45 on substitution and pairing, 3.4 on constraints; every ranked model sits well above that last line.
One substitution item: replace boursin cheese. A high-scoring model picks caciocavallo, fromage blanc, and grana padano for 100. A low-scoring model picks fromage blanc, quail egg, and goose egg for 15. Continuous scores beat exact match here, because two non-optimal portfolios can differ by a lot of culinary utility.
The lesson is for evaluation more than for chefs. An executable environment, a complete rectangular matrix, frozen score maps, and multiplicity-controlled paired tests transfer to any domain that has a simulator and no single gold string. The maps are also a training signal: supervise the optimum, learn from pairs, or run offline policy optimization against a bounded reward. This paper does not train. It does release the environment, the reward, and the anchor IDs, which is enough for a clean next experiment: train on development maps, test on held-out anchors. Optimizing the public test map would measure memorization.
For leaderboard readers: Grok 4.6 and Gemini 3.1 Pro differ by 0.1, with fully overlapping simultaneous intervals. Calling that a win overstates the evidence. Command R+ has rank interval [27, 27]. That bottom slot is resolved.
The authors are direct. The score is agreement with one Epicure release, not universal human taste. Within-task min-max makes 0–100 comparable and discards absolute utility. Tasks are constrained three-item selection, not full recipes, sensory execution, safety advice, or long-horizon kitchen planning. The common core estimates performance on tasks every endpoint completed; hard-to-parse or refused items are out. Results bind particular routes and collection dates. The dense reward surface could support learning; this paper does not test whether optimizing Epicure reward transfers to human cooking.
Josef Chen is an independent researcher and Erim Hayretci is at Imperial College London. Task text and rewards both come from the same Epicure embedding. If frontier models saw similar ingredient co-occurrence in pretraining, part of the score may be corpus overlap. There is no reported correlation with human chef preference. Constraint tasks have only four distinct scores, closer to feasibility classification than to the graded semantics of the other two families.