Local agent benchmarks yield 3000 ground-truth samples and a DPO+LoRA training plan
julianharris · x · 2026-10-04
julianharris is running dozens of long end-to-end app builds (e.g. "Build an MVP Miro Clone") to benchmark local AI across three hardware types, hunting for consistent model behavior patterns (likely Qwen's). Key finding: models think hardest immediately after a test failure — a signal he wants to cut further, potentially via DPO + LoRA. His benchmarks have already generated 3000 ground-truth sources, a "production flywheel" where the model creates its own training data. Project and benchmark data are on GitHub.
More from coding & agent
- Gradio and HF launch hackathon: reproduce NeurIPS 2026 papers with AI agents — Gradio · 2026-11-19
- Ramp SWE-Bench v2: top four coding models within points, but spend varies wildly — charliermarsh · 2026-11-19
- LangChain's deepagents hits 50M downloads, now the industry-standard agent harness — LangChain · 2026-11-19
- NVIDIA open-sources SkillEvaluator to vet AI agent skills before deployment — anthara_ai · 2026-11-19
- Resolve.ai publishes SRE-1 paper on AI agents for production incident investigations — iceman84 · 2026-11-19
- GPT-6.2 in Codex: split reviews hinge on your first line, says thread with 5 prompts — alexcovo_eth · 2026-11-19