Qualcomm demos an on-device decision model routing AI requests in 8.6ms on NPU

carrycooldude · x · 2026-11-17

A Qualcomm dev account highlights Lucas da Silva's experiment with open-source Laya: a small decision model that inspects every AI request first, running on the Snapdragon X Elite NPU with no quantization or retraining. It detects private data, triages requests, and decides whether work stays local or goes to the cloud.

Results: 8.6ms per decision vs 36ms on CPU, with 20x+ less energy — a promising pattern for local-first routing before bigger models step in.

Original post →

More from coding & agent

coding & agent channel →