Dev compiles incident-triage SOP into a fine-tuned 3B model: 93% accuracy at 8.4x lower cost

arpit_bhayani · x · 2026-10-24

Arpit Bhayani ran a contrarian experiment: instead of prompt engineering, he "compiled" a 3,500-token incident-triage SOP into a QLoRA-tuned Qwen2.5 3B model, moving the procedure from the system prompt into the weights.

Results across 500 test scenarios:

The fine-tuned 3B hit 93.0% end-to-end success, beating Gemini 2.5 Pro with cache (92.0%) and raw API (90.0%); the untuned base scored just 20%;
Input tokens per incident dropped from 5,200 to 450 (-91.3%);
Cost per 1,000 incidents fell from $1.85 (cached) to $0.22 — 8.4x cheaper.

The key trick was loss masking: alerts and tool outputs are masked so the model learns only assistant decisions, not telemetry noise. He acknowledges the tradeoff — any workflow change requires retraining. The workflow-slm repo is open on GitHub with a detailed write-up.

Original post →

Dev compiles incident-triage SOP into a fine-tuned 3B model: 93% accuracy at 8.4x lower cost

More from coding & agent