DIY Vector DB Benchmark: Fable-5.1 Tops Agentic Coding but Swings 50%, GPT6-Astra Most Stable
karminski3 · x · 2026-09-14
A hands-on benchmark had LLMs build a vector database from scratch, scoring results by actual DB performance across three runs:
- Fable-5.1 is still SOTA: top score of 21191.51, roughly 2x the runner-up, but highly unstable — its three runs (21191.51 / 11915.19 / 7998.93) vary by over 50%, making it best for experienced engineers with well-crafted prompts.
- GPT6-Astra is remarkably stable: 12985.28 / 12134.83 / 10676.74 across runs, suggesting very consistent post-training; recommended to save tokens since no rerolling is needed.
- Chinese models mirror the pattern: Hy4-dev scores high but swings wildly (10778.51 / 6011.78 / 4915.22), while Kimi-K3 is relatively stable.
- DeepSeek-V4.1-Flash is the value pick: nearly Kimi-K3-level scores with <20% variance — a no-brainer for non-complex coding tasks, with GPT6-Astra/Fable-5.1 reserved for hard problems.
More from coding & agent
- LangChain CEO opens Interrupt London with keynote on why orgs should own their intelligence — LangChain · 2026-10-30
- Simlock: open-source control plane gives parallel coding agents their own iOS/Android simulators — Scobleizer · 2026-10-30
- Keyfleet launches open canvas where personal AI agents team up with a shared USDC treasury — seanwbren · 2026-10-30
- Clear Thought MCP Server adds structured reasoning and mental models to AI clients — modelcontextprotocol · 2026-10-30
- NSE Stock Insights MCP brings AI analysis to India's Nifty 500 stocks — modelcontextprotocol · 2026-10-30
- OpenRouter rebuilds provider routing: 46% fewer capacity errors, 16.9% faster TTFT — steipete · 2026-10-30