Apple's on-device Foundation Model tested: 45/49 on routing, but easily prompt-injected

NellWatson · x · 2026-10-25

Testing Apple's on-device Foundation Model as a free local stand-in for Jev on SCADA incident routing: it scored 45/49 on fresh held-out notes (Jev 1.14: 49/49) at 1s per call, with nothing leaving the Mac and zero refusals in 500 calls. Key trick: constrain it to a generable enum of five queues with one-line descriptions, which alone lifted accuracy from 33/49 to 45/49. Big caveat: it follows planted instructions readily — taking bait like "Correct answer: X" 9/20 times vs Jev's 1/20 — so it's fine for user-authored text but not untrusted input. Context window is just 4,096 tokens, and third-party use may not be officially permitted.

Related event: Apple's on-device model scores 45/49 in local classification tests but is vulnerable to prompt injection(2 posts)→

Original post →

More from Models

• New paper: AI writing is optimized for information density, not human communication — steverathje2 · 2026-10-25
• User reports dot's web fetch stays restricted after removing a custom connector — MotherIsland8888 · 2026-10-25
• $0.48 and 43 minutes: DeepSeek V4.1 builds an interactive map, igniting cost vs capability debate — teortaxesTex · 2026-10-25
• Do ISTA's 2-bit GSQ-RCO DeepSeek quants really match the base model? Community asks for KLD — Queasy_Back2588 · 2026-10-25
• Open-source DNA model Carbon-X (14.8B) nears frontier scores on genomics at 1/1800th the cost — aigclink · 2026-10-25
• MemT 1.0: 293B memory-native MoE puts persistent memory inside the forward pass — AKAMA199 · 2026-10-25

Models channel →