Probe-Guided Fine-Tuning for Alignment: ELLIS Team Open-Sources LoRAs and Detectors
maksym_andr · x · 2026-10-20
The AI Safety and Alignment Group at ELLIS Institute Tübingen and MPI-IS released LoRA adapters and detectors from their paper on alignment via training against probes without losing monitorability.
- Method: probe-guided fine-tuning — instead of rewarding outputs that look aligned, probes detecting undesired properties in activations serve as direct training signals, using either a single linear probe or a non-linear polytope of K=16 probes, frozen or continuously updated
- Experiments: harmlessness trained on Mistral 7B Instruct v0.1 and abliterated Llama 3 8B Instruct; honesty on Qwen3-14B
- Release: adapters at all Pareto-front steps plus detectors; e.g. the continuously-updated Mistral run reaches StrongREJECT 0.03 with utility 0.96 at step 47
More from Safety
- Cloudflare MCP Server adds step-up auth: agents request scopes on demand — irvinebroque · 2026-10-20
- UK reportedly weighs EU's CADA law that would bar its government from non-European AI models — NathanpmYoung · 2026-10-20
- Malta pays €4 million for Microsoft Copilot through 2028 with unchecked lock-in — YvesMulkers · 2026-10-20
- Bayesianism is dangerously overused in nuclear attack computers, researcher warns — NathanpmYoung · 2026-10-20
- Devs are reinventing SIEM from scratch — next stop, the AI SOC — thedealdirector · 2026-10-20
- Google Cloud upgrades VPC Service Controls with folder-level perimeters for agent traffic, free — rseroter · 2026-10-20