Blind 3-provider LLM benchmark: GPT-6.1 Sol beats Claude 9-0, structured output dialects differ
WorthRow3022 · reddit · 2026-11-14
Sharing learnings from adding OpenAI and Gemini alongside Claude in the open-source CloudArchitect Lite (plain-English app description → AWS architecture diagram):
1. "Structured output" is not one standard. OpenAI strict mode rejects descriptions next to $ref; Anthropic doesn't support min/max on integers; and valid JSON can still be semantically wrong (one reviewer returned two reviews for one design, none for another). What works: one Zod source schema, a small adapter per provider, and re-validating every answer in code.
2. A blind review panel, one reviewer per provider. Six anonymized models scored on a 7-dimension rubric with the median of three cross-provider reviewers: GPT-6.1 Sol beat Claude Sonnet in all 9 pairs; even the Claude reviewer preferred Sol 4–3. Reviewers lean toward their own family only on near-home pairs. Deterministic rule checks caught what reviewers missed: the best-reviewed model left a public API without stated auth in 6 of 15 designs. Limits: 9–12 designs per model, model reviewers, $12 total cost.
More from coding & agent
- Your company 'second brain' is just silos with an AI on top — alex_verem · 2026-11-14
- Modal Endpoints land in Nous Research's Hermes Agent via one-line plugin install — AAAzzam · 2026-11-14
- Cmpnd releases dspy-wsp: $5-10 of GPT-6 edits model weights, lifts every decision model tested — dbreunig · 2026-11-14
- AmphoraDB launches serverless databases for agent swarms, paid per statement in stablecoins — banteg · 2026-11-14
- Ex-quant's local AI agent negotiated 1,000+ influencer outreach in 9 days — Shruti_0810 · 2026-11-14
- With 90% AI-generated code, Limestone gates merges on its V.U.E. three checks — alex_verem · 2026-11-14