Running a 3B roleplay LLM fully on iPhone: what actually keeps it in character

Low-Future-9387 · reddit · 2026-10-09

The developer of Castmates shared detailed engineering notes on running a 3B roleplay model entirely on-device (Impish Llama 3B + rank-16 LoRA, Q4KM, 2GB).

Setup

llama.cpp with Metal, all layers offloaded, q80 KV cache (60KB/token), fully offline
Memory tiers: 4GB phones cap at 4096 context, 6GB get 6144, 8GB get 8192; 6 tok/s measured on A18
The 2GB download is the biggest drop-off; Background Assets pre-loads it before first launch

Findings on staying in character

Context window size alone did nothing — the history-trim budget was the binding constraint. Scaling trim to 0.68×nctx took long-conversation fact recall from 25% to 75%; verbatim history beat a lossy summarizer
BM25 retrieval re-injected superseded facts (says-stale +20.8pp); filtering hits whose rare entities still appear in the live window cut it by 18.8pp
Prompt wording tweaks measured as zero without N=16 same-seed controls; placement beats wording — a scene direction lands 0/16 in the reminder slot but 16/16 as its own block after the final user turn
Code guards (rerolls on person drift, invented names) beat prompts for 3B habits; thinking mode just added latency and noise

The model still can't override facts it can read in context — a hard ceiling. Every claim came from fixed-seed lab runs of the production prompt pipeline, not vibes.

Original post →

Running a 3B roleplay LLM fully on iPhone: what actually keeps it in character

More from coding & agent