AWS guide: deploy llama.cpp decision models on SageMaker, GPU cuts tail latency 32x
AWS ML Blog · rss · 2026-10-12
An AWS ML Blog post shows how to deploy decision models—routing, agent action selection, policy calls needing scores over fixed options—on SageMaker AI using the llama.cpp DLC.
Two paths: the standard DLC serves OpenAI-compatible chat completions; a structured-decision path builds a child image with SemIf (formerly OpenJev) that skips answer-token generation and directly returns per-option probabilities (including optionlogits) for thresholds, fallbacks, and policy checks.
Benchmarks (231 JevBench tasks, Qwen3.5-4B Q4KM): CPU (ml.c6i.8xlarge) hit p50 1.715s / p95 19.3s vs. GPU (ml.g6e.xlarge, full offload to an L40S) at 0.352s / 0.599s—4.9x faster at p50 and 32.2x at p95, with 77.5% vs 77.9% accuracy (runtime variance). Smaller 2B/0.8B models cut latency but accuracy fell to 66.7%/58.4%, so 4B stays the quality-first default.
Guidance: CPU for background/low-throughput work; GPU for interactive or tight tail-latency targets; shared-state scoring reuses the prompt prefix; keep authorization and deterministic rules outside the model.
More from coding & agent
- rakyll: attacked last year for saying coding agents move too fast, still worth watching — rakyll · 2026-10-13
- Instead of compaction: letting Claude Code wipe its own context and resume from checkpoints — Motiv31 · 2026-10-13
- Amp Adds Interactive Views, Letting You Mock and Test UIs Right in Chat — iannuttall · 2026-10-13
- Using Muse Wonder's CUA as a verifier agent for self-improving code generation — EdwardSun0909 · 2026-10-13
- Meta's new Wonder model praised for finding security bug other agents missed — alexandr_wang · 2026-10-13
- Claude effort levels in practice: medium suffices for 80% of tasks — EricBuess · 2026-10-13