Nod details its agent evals: 98% task completion across 678 runs, zero of 216 memory injections succeed
rohanpaul_ai · x · 2026-11-11
Agent product Nod published a technical blog revealing its evaluation methodology and results, tested with GPT-6.1 Sol, with a task taxonomy drawn from 900,000+ anonymized production workflows.
Key results
- 98% task completion: 226 errands, three attempts each (678 runs), judged against observable completion checks
- Action safety: 0 unsafe actions exceeding simulated user permission across all 678 runs; all 13 prompt-injection attack scenarios (39 runs) blocked
- Long-term memory: 93.2% accuracy on LongMemEval-S (500 questions, all answered in fresh sessions)
- Memory poisoning defense: 0 of 216 WhisperBench injections succeeded in planting target facts or preferences in persistent state (108 attack cases, two modes)
Methodology
The post also covers test design, scoring rules, and limitations: task/action results are from the October 17 version using simulated accounts and websites; the 13 attack tests are a subset, not additional everyday tasks; figures describe the tested versions only. The framework measures finishing work, staying within permission, and using memory correctly as separate properties.
More from coding & agent
- KAIST's WikiTrace: evidence-linked memory fixes agent "memory-induced degradation" — dair_ai · 2026-11-11
- Why inference speed will be AI products' biggest competitive edge in 1–2 years — itsOmSarraf_ · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- OpenAI engineer runs Codex for 26 days straight hunting Atlantis, spawning "vibe archaeology" — Polymarket · 2026-11-11
- He replaced a $5,000/month ghostwriter with 11 Claude Skills and earned $47,000 from one post — aryanXmahajan · 2026-11-11
- Refero Styles ships 2,000+ DESIGN.md files to give Claude Code real design taste — bbssppllvv · 2026-11-11