Skip the GB-heavy abliterated models: a tiny rank-1 LoRA uncensors at runtime
returnity · reddit · 2026-11-16
A Reddit PSA: stop downloading hundreds of GB of abliterated models—most abliteration techniques are rank-1 ablations whose refusal projection fits in a tiny LoRA you can apply at runtime to a quant you already have.
Method and data:
- After comparing heretic, Orcarouter, and huihui v1/v2, the author built a LoRA from huihui v2. On Aider Polyglot it even outscored the base model, with fewer tokens and no runaway loops.
- Quality checks: Wikitext KLD of 0.040, 93.7% same-top-token rate, perplexity statistically identical to base; Jailbreak Bench refusals (thinking on) dropped from 93/100 to 3/100.
- The rank-1 refusal direction and per-layer strength were extracted with Claude Opus 5.5 via weight-space programming, diffing huihui weights against the Unsloth quant—after dodging safety guardrails by renaming files and framing it as an "anti-slop LoRA experiment."
Usage (llama.cpp):
bash
llama-server -m Qwen3.8-Flash-Next-UD-Q5KXL...gguf \
--lora HUIHUI-V2-...-LoRA-F32.gguf --lora-init-without-apply -fa on --jinja
Adding a lora field per-request enables on-demand switching: serve the unmodified model, and only enable the LoRA on refusal and resend. The author suggests a pi extension to automate this (at the cost of a full re-prefill). The technique also works in Strata and ds4/dwarfstar.
More from Infra
- ETH's Hoefler previews SC26 keynotes: weather-model scaling laws and AI infrastructure spend — thoefler · 2026-11-16
- NVDA earnings seen as catalyst as tech rally hits resistance and retest of lows looms — pdamodaran · 2026-11-16
- Cerebras docs CS-4 engineering film: 4th-gen wafer-scale system is 2x faster than anything it has served — AccBalanced · 2026-11-16
- AI buildout is a real estate business that happens to compute, not a tech hype cycle — brucemacv · 2026-11-16
- GPU Shortages Push Modal to Tap More Than 30 Small Cloud Providers — AccBalanced · 2026-11-16
- Data center construction stocks reportedly shed $4.2T in market value since June — Polymarket · 2026-11-16