DIY Vector DB Benchmark: Fable-5.1 Tops Agentic Coding but Swings 50%, GPT6-Astra Most Stable
karminski3 · x · 2026-09-14
A hands-on benchmark had LLMs build a vector database from scratch, scoring results by actual DB performance across three runs:
- Fable-5.1 is still SOTA: top score of 21191.51, roughly 2x the runner-up, but highly unstable — its three runs (21191.51 / 11915.19 / 7998.93) vary by over 50%, making it best for experienced engineers with well-crafted prompts.
- GPT6-Astra is remarkably stable: 12985.28 / 12134.83 / 10676.74 across runs, suggesting very consistent post-training; recommended to save tokens since no rerolling is needed.
- Chinese models mirror the pattern: Hy4-dev scores high but swings wildly (10778.51 / 6011.78 / 4915.22), while Kimi-K3 is relatively stable.
- DeepSeek-V4.1-Flash is the value pick: nearly Kimi-K3-level scores with <20% variance — a no-brainer for non-complex coding tasks, with GPT6-Astra/Fable-5.1 reserved for hard problems.
More from coding & agent
- Skills Aren't Docs: Meta-Programming LLMs to Generate Micro-Skills for Arbitrary Invoice Formats — karminski3 · 2026-09-14
- A new law for AI coding: never generate more code than you can fix in one sitting — cthechartreuse · 2026-09-14
- Open-source Claude skill automates App Store competitive research — iamdsvs · 2026-09-14
- When three AI agents want to change the same landing page, who owns it? — OwlZealousideal4779 · 2026-09-14
- Cognition Recruits Devin Ambassadors Across Nearly 40 Cities Worldwide — cyrus_zei · 2026-09-14
- Chroma Lingua: agents get Plutchik's Wheel of Emotions as a skill, rendered as subtly animated text — RileyRalmuto · 2026-09-14