Skip LoRA: fine-tune 4K-dim token embeddings with evolutionary search instead of 9B parameters
cephaloform · x · 2026-10-09
- cephaloform shares an unconventional fine-tuning approach: instead of touching model parameters, fine-tune the embedding representations of tokens in the prompt using evolutionary algorithms.
- Rationale: a token embedding is very low-dimensional (4K vs. 9B parameters), a search space evolutionary methods can actually work in.
- Claims it doesn't break kernels the way LoRA can, and that back in the day it even outperformed full model fine-tuning on some tasks (links attached).
It's a middle path between prompt engineering and parameter tuning — moving "what gets learned" from weights to input embeddings, with low compute cost and implementation barrier. Worth a look for developers wanting cheap local model customization.
More from Research
- Fields medalist: OpenAI solving 350 major math problems felt like being crushed by trucks — soumitrashukla9 · 2026-10-09
- Dev uses Astra to mine NASA datasets for planets, reports a promising lead — BLUECOW009 · 2026-10-09
- AgentGarten trains agents in rendered code worlds, learning in 4 rounds vs millions for RL — MirroS-Lab · 2026-10-09
- V-CoLA: training-free vision token compression keeps 99.5% performance at half the tokens — ATH-MaaS · 2026-10-09
- Evoke open-sourced: 30M-param model in Postgres matches 0.6B embedding model on recall — bdsqlsz · 2026-10-09
- University of Toronto's Artificial Consciousness Initiative hires postdoc in AI philosophy — KathleenACreel · 2026-10-09