oMLX ports antirez's ds4 directional steering, cuts Qwen3.8 verbosity ~50% with ~0 NLL cost
Undici77 · reddit · 2026-11-20
A Redditor ported antirez's ds4 directional steering idea into the local inference engine oMLX and benchmarked it on Qwen3.8 Flash Next.
How it works: a "pack" learns one direction per layer from paired prompts (e.g. succinct vs verbose requests); at inference a hook shifts each layer's attention-branch output along that direction with live-adjustable strength — no fine-tuning, and nothing changes unless a pack is set.
Results (MacBook Pro 14" M4 Max 128GB, oQ4e+MTP, 100-pair verbosity pack, temp 0):
- Strength -0.5: answer length -24-30%, KL on neutral text 0.021, NLL -0.3%
- Strength -1: length -46-50%, KL 0.045, NLL +0.2%
- Lower repetition, valid JSON unchanged
- Cost: +0.2–0.4% (noise-level) when applied to prompt + first 32 tokens; 5% when applied to every token
Code is on the steering-experimental branch, with a call for votes to merge upstream.
More from Infra
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Nvidia's rivals are vying to be the 'Central Bank' of AI, says investor — GavinSBaker · 2026-11-21
- Benchmarking 11 open models' post-GQA attention: up to 139x less memory at 128K — bariskasikci · 2026-11-21
- Anthropic files for Nasdaq IPO — why you should keep a local open-source LLM as backup — alex_verem · 2026-11-21
- Microsoft launches MAI Licensed Weights; Purdue researcher asks to run them on own hardware — marlene_zw · 2026-11-21
- Looped CED cuts KV-cache memory 36.8x on 552B model, but encoder loops cost compute — teortaxesTex · 2026-11-21