Compile Your LLM Workflow Into Small Model Weights: A LoRA Hypothesis Worth Testing
arpit_bhayani · x · 2026-10-21
Engineer arpitbhayani proposes "compiling" mostly-deterministic LLM workflows into small fine-tuned model weights instead of paying for long prompts on every call.
The approach
- Write the workflow (e.g. support ticket triage: parse → extract order ID → look up order → apply refund policy → draft reply) as an explicit step graph
- Add a validator per step, run thousands of realistic inputs, keep only fully passing traces
- LoRA fine-tune a small model on those traces so the procedure lives in the weights; runtime prompts shrink to raw input + workflow ID, with constrained decoding ensuring only legal transitions and valid tool calls
- A thin runtime executes tools, validates results, and feeds them back; deterministic parts (arithmetic, thresholds) stay in plain code, the model handles only fuzzy steps
Caveats: weights alone don't guarantee determinism (temperature 0 + constrained decoding help), validators are essential, workflow changes require retraining, and out-of-distribution inputs are where small models fail first.
Unbenchmarked — offered as a hypothesis worth testing. Only pays off when the procedure is fixed, inputs are messy, and volume is high.
More from coding & agent
- Dev built self-maintaining AI agent tools all year just to play GTA 6 in peace — rudrank · 2026-10-21
- Artificial Analysis opens up its Coding Agent Index: 39 configs, 3 benchmarks, full methodology — ArtificialAnlys · 2026-10-21
- Devin Fusion's dual-model setup scores 68 on Coding Agent Index at 31% lower cost than Claude Code — ArtificialAnlys · 2026-10-21
- Plausible MCP Server lets AI models query web analytics via natural language — modelcontextprotocol · 2026-10-21
- Confluence MCP connector wraps Confluence Cloud REST API v2 with OAuth — modelcontextprotocol · 2026-10-21
- Why agents need an 'Agent GitHub' for versioning, permissions, and memory — yangyi · 2026-10-21