Runner trains LoRA directly on quantized GGUFs — no PyTorch, no FP16 base needed
ZenZombie117 · reddit · 2026-10-14
A Redditor shares xyntetik-runner, an open-source tool that trains LoRA adapters directly on the quantized GGUF weights you serve — no PyTorch, transformers, or FP16 parent model required. On CPU, the training forward pass is the serving forward pass, so the adapter is trained on the exact weights it will run on.
Usage (commands included):
- --train data.l --train-steps 200 --lora-rank 8 trains adapter.gguf on a frozen Q4KM base, logging sha256 of base/data/adapter, seed and config
- --lora adapter.gguf --serve serves the adapter beside the base
- --merge-lora merged-Q80.gguf --quant q80 folds it into a standalone file for any GGUF runtime (8-bit+)
- -f eval.txt --score gives teacher-forced logprobs for evals
Caveat: 4-bit adapters "disappear" when merged into weights, so the tool refuses such merges by default. Author validated on Qwen3-4B Q4KM with before/after tool-call scores, noting results may not generalize. Very practical for local fine-tuning and on-device deployment.
More from Infra
- ECDSA.fail quantum challenge hits 893 qubits x 2.2M Toffolis, 25% cheaper than Google's best — jedisct1 · 2026-10-14
- Llamaforge v0.16: a control panel for upstream llama.cpp with local agents whose quotes are code-verified — Sleepybear2611 · 2026-10-14
- Questioning OpenAI's Decisions API: are cloud round-trip latency and privacy viable in production? — WhichCardiologist800 · 2026-10-14
- Kev-4B Decision Model Goes Live in Beta on Phala with NVIDIA Confidential Computing — bgmshana · 2026-10-14
- Zuckerberg: Meta committed hundreds of billions on AI infra before knowing it would work — GaryMarcus · 2026-10-14
- NVIDIA's 400G/lane optical spec puts tougher laser demands on CPO — jwt0625 · 2026-10-14