Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference
Christos Koutsiaris
cs.IR, cs.AI, cs.CL, cs.LG
2026-08-21
A 160M hybrid keeps attention in 6 of 18 layers. On 59.9B tokens it scores 47.31 vs a 42.20 bar, and is 1.76x faster than a matched dense twin at 2048-token CPU decode.
Small language models are usually designed like their large siblings, then squeezed onto a CPU. That order is wrong for the one-user, one-token-at-a-time regime.
There is no batch to amortize weight loading. Every generated token streams the whole model through the memory hierarchy, so throughput is bytes per token, not FLOPs. Attention adds a second tax: a KV cache of past keys and values that grows with the conversation and is re-read at every step. On a GPU serving large batches the tax is tolerable. On a CPU at batch size one, it is what users feel as lag in a long chat.
If most layers carried a constant-size state instead of a growing cache, decode cost would flatten with context length. That is the lever this paper pulls.
Daedalus-150M has 160.49M parameters in 18 blocks at width 768 and context 2048. Only six blocks run full attention. The other twelve use a depthwise convolution with kernel length 3, each channel sliding on its own. The stack is interleaved CCCC A CC A C A C A C A CC A C, with attention at layers 4, 7, 9, 11, 13 and 16, spread through depth rather than clustered so retrieval sits at several levels of representation.
A convolution block projects the input three ways into B, C and x, applies a depthwise conv1d to the gated x, then projects the C-gated result back. Because the kernel is length 3, the recurrent state is always two timesteps, independent of how long the conversation has grown. B and C supply the input-dependent behavior a fixed kernel lacks. Running a convolution layer costs the same at token 2000 as at token 2.
The six attention layers use grouped-query attention, 12 query heads sharing 4 KV heads, which cuts their cache by another factor of three. The 49,152-entry embedding is tied, so the 37.7M-parameter table is stored once and still takes 23% of the model. The feed-forward inner width is 2048 (2.67x the model width, not the usual 4x), moving parameters out of the fattest, most bandwidth-hungry tensors.
The control is a dense twin matched to 0.5%: 24 all-attention layers at width 640, FFN 2304, 161.25M parameters. Both arms saw the same data and schedule for 5B tokens, fully decayed. The winning rule was written down first: a 0.5% floor on validation bits-per-byte. Short convolution was chosen over a selective-scan state-space layer because it maps onto existing CPU kernels with no new operators. The target runtime is llama.cpp Q40. Mixture-of-experts was rejected at design time: no sub-1B precedent, and the runtime's mix path hardcodes a vendor's gating, so the model would not export. Distillation was cancelled because teacher logits needed about 288 GB on a 250 GB disk.
The full run trains on a ten-source English mix, 16.93B unique tokens, a 59.9B budget, about 3.5 epochs, each source capped at four repeats, on one RTX 5090.
Validation bits-per-byte on the finished model is 0.8685. The five-task mean is 47.31 against a bar of 42.20, which is GPT-2 124M on the same harness. Peers were re-scored locally; MobileLLM could not be run on that harness, so its published 46.3 is quoted as-is.
| Model | Tokens | 5-task |
| Daedalus-150M | 59.9B | 47.31 |
| MobileLLM-125M | 1T | 46.3 (published) |
| GPT-2 124M | — | 42.2 |
| OPT-125M | 180B | 42.1 |
| GPT-neo-125M | 300B | 41.9 |
| Pythia-160M | 300B | 41.0 |
| Peer-135M | 2T | 51.2 |
Per task: PIQA 65.78, ARC-Easy 50.42, WinoGrande 50.04, HellaSwag 37.93, OpenBookQA 32.40. WinoGrande sits on its 50% chance floor; at this size that task measures almost nothing. The 2T Peer-135M remains 3.9 points ahead, a gap conceded before training: quality was traded for decode speed at a fixed size. The same architecture trained on only 5B tokens already scores 44.7, which clears 42.2.
The 5B-token ablation isolates the architecture. Hybrid valbpb 0.9104 versus dense 0.9178, a 0.81% win that clears the 0.5% floor. Five-task means 44.68 versus 44.82, a 0.14-point gap of about 0.24σ, with tasks swapping places. Treat that as noise. The 4-bit file is 95.56 MiB versus 101.62 MiB, 6.3% smaller.
CPU decode (Q40, 8 threads, 128 generated tokens) is the claim, and the shape of the two curves is the thesis:
| Context | Hybrid tok/s | Dense tok/s | Ratio |
| 0 | 1111.9 | 922.8 | 1.20x |
| 512 | 960.3 | 664.4 | 1.45x |
| 2048 | 739.3 | 420.3 | 1.76x |
The gap is near nothing on an empty context and widens with length. Against an external 135M model the same signature appears: 1.06x at depth 0, 2.08x at 2048 (648.6 vs 312.4 tok/s). Byte accounting says the hybrid moves half the KV traffic (6144 B vs 12288 B per context token) and should be 1.17x at 2048. The measured 1.76x leftover is attributed to softmax latency and the twin paying per-layer overhead 24 times instead of 18. A merely thinner model would be faster by a constant. This one is not.
The audience is anyone running a local small model on a commodity CPU. Long chats, documents stuffed into the prompt, and file processing sit at the right-hand end of that table. Short independent prompts into an empty context see almost none of the gain. The shipped file is 95.56 MiB; at 2048 tokens the KV cache adds about 12.6 MB, so a single-user session fits under 128 MB and can live inside an application process.
There is no new operator. The move is treating the ratio of cache-free to attention layers as the main design knob for batch-size-one CPU decode, then isolating that knob with a pre-registered twin. Griffin-style hybrids live in the same family. The difference here is a convolution short enough for stock kernels, with the ratio set by a memory-traffic argument rather than GPU habit.
This is an incremental architecture paper pinned to a deployment constraint, not a new small-model leaderboard.
Quantization-aware training died on a non-finite loss at step one and was turned off, so the 4-bit penalty is the full post-training hit: about 6% perplexity (9.18 at half precision versus 9.75 at Q40), against 2.5% at the 5B-token scale. About 47.9% of convolution channels contribute nothing, roughly 13.6M inert parameters, 8.5% waste. Export-time pruning was rejected by llama.cpp's shape checks (768 expected, 640 received). The 49,152 vocabulary was inherited from a cancelled distillation plan; scaling laws put a 150M model nearer 24k-32k, and embeddings currently hold 23% of the parameters. Mixture L1 drift landed at 10.42 against a pre-set limit of 10.0. The last 8% of training resumed from a checkpoint with optimizer momentum zeroed, the data cursor reset, and a corpus 0.42B tokens smaller.
Every number is one seed. The 0.81% ablation margin is not a confidence interval. The model is English-only. Speed is measured at the trained 2048-token context, generation only, no prefilling. WinoGrande is padding the five-task mean.