Write-up: compiling a 3,500-token workflow SOP into a QLoRA-tuned 3B model

arpit_bhayani · x · 2026-10-24

This is Arpit Bhayani's detailed write-up of the workflow-slm experiment: delete the 3,500-token SOP from the prompt and use QLoRA fine-tuning on Qwen2.5 3B to bake the full incident-triage procedure (parse alert → check metrics → check deploys → read logs → root cause → draft status update) into the weights.

Full benchmarks included: the tuned 3B reaches 93.0% end-to-end success vs 92.0% for cached Gemini 2.5 Pro and 90.0% raw API; input tokens drop 91.3%; cost per 1,000 incidents is $0.22, 8.4x cheaper than the cached setup. The critical technique is loss masking — alerts and tool outputs are excluded from the loss so the model only learns the assistant's decisions. Repo: arpitbbhayani/workflow-slm.

Original post →

Write-up: compiling a 3,500-token workflow SOP into a QLoRA-tuned 3B model

More from coding & agent