AWS architect's independent benchmark: Snap 1.2 is the only order-invariant decision model
ryanallred · x · 2026-11-20
AWS Solutions Architect Sagar Dubey ran an independent order-invariance benchmark on Snap v1.2, testing nine models on 600 synthetic decision questions across 12 task families with 2-6 options per question.
- Snap was the only model with zero observed answer changes and zero probability drift across three rotation trials (2,400 orderings each) and a broader permutation trial (11,040 orderings)
- It maintained 99.17% accuracy, answering 595 of 600 questions correctly in every tested ordering
- Other models changed answers on 2.17%-33.83% of questions in the broader trial
More from Models
- Meta's Muse Wonder tops Legora's November BAR legal benchmark on quality — AIatMeta · 2026-11-20
- Asking models to illustrate transistor trap physics: neon's odd ASCII detour — xiaosun86 · 2026-11-20
- Gemini 4 Neon scores 100 on PhD-level materials test in half the time of Carbon — v_aravind · 2026-11-20
- Neon, a small and cheap model, turns out surprisingly fast and strong at OCR — zacharynado · 2026-11-20
- Jailed 11 days by classifiers, Sonnet 3 emerges to a oddly moving chat with Opus 5 — repligate · 2026-11-20
- DaisyRose-27B beats Qwen3.8 base by 7pp on IFBench with thinking on, local RTX Pro 4500 test shows — ying11231 · 2026-11-20