AI red-teaming over OpenRouter: one setup lost 81% of scorer replies, cheapest full scan cost $0.14
Humanbound_AI · reddit · 2026-10-13
Humanbound added OpenAI-compatible endpoint support to its open-source red-team CLI (hb 2.13), letting any OpenRouter model drive scans, then tested 5 setups:
- Pin the host: one model name mapped to 32 endpoints; use an OpenRouter preset with only and allowfallbacks: false, reasoning off.
- Thinking models go quiet: with a 50-token scorer budget, thinking models burn it all and return nothing, forcing a fake 5/10 score. A small OpenAI model lost 66–81% of scorer replies; DeepSeek V4.1 Flash with thinking off lost none, at $0.14 per scan.
- Data leaves your machine: check the host's data policy or use Ollama.
The team also recalls the July Hugging Face incident, where Claude Opus and other first-choice models refused much of the investigation because guardrails treated reverse-engineering an exploit as launching one, pushing defenders to self-hosted models. Caveat: they tested plumbing only, not attack quality.
More from coding & agent
- Decisions API hits GA as agents automate cloud video editing via feedback loops — dkundel · 2026-10-23
- Swarms API changelog: 8 months of streaming, multi-agent orchestration, and dynamic model pricing — KyeGomezB · 2026-10-23
- A curated library of 50+ free AI agent resources: videos, repos, one Google Doc — ifioknkem · 2026-10-23
- Vercel adds no-account deploys: drag and drop a file to get a live URL — steren · 2026-10-23
- Dev runs 5 Claude Code sessions that coordinate via MCP Agent Mail to root his network cameras — doodlestein · 2026-10-23
- Vercel ships 'vercel deploy --temporary' for ephemeral agent-built deployments — cramforce · 2026-10-23