Apple Neural Engine: Architecture, Programming, and Performance
Spencer H. Bryngelson
cs.AR, cs.OS, cs.PF
2026-06-21
Reverse-engineered ANE from datapath to firmware. M1: ~12 fp16 TFLOP/s, ~0.37 pJ/FLOP; 256-ch 3x3 conv is 3.8x faster and 9x more efficient than GPU. Direct path is unsupported.
The Apple Neural Engine has shipped in every Apple SoC since the A11 in 2017 and the M1 in 2020. Apple's active installed base passed 2.5 billion devices in early 2026, and most of those chips contain one. It runs on-device vision, speech, and language models. Among the programmable engines on an Apple chip it is the most opaque: no public ISA, no driver interface, and no documented way to confirm a computation even ran on it. Apps reach it only through Core ML, where computeUnits is a placement hint. A cost-driven planner splits the graph across CPU, GPU, and ANE, and the caller is never told which device ran each segment.
This roughly 300-page guide reverse-engineers the engine from the fp16 datapath up to firmware. Two lines of evidence check each other: direct measurement on Apple silicon, and static decompilation of the private runtime, compiler, kernel driver, and firmware. The primary host is the M1 (internal name H13); a second physical chip, the M5 (H17s), checks the cross-generation claims, with an M2 filling the middle. The measured direct route is released as the open-source ANEForge runtime. It is callable from ordinary user space, undocumented, unsupported, and version-fragile. It is for measurement and research. Shipping software still belongs on Core ML.
The entry point is the private Espresso runtime Apple's own dispatchers use. It lowers a graph to the engine's program format, loads it, and drives an execution stream, with no placement planner and no entitlement for the operations the compiler accepts.
Four techniques converge. User-space counters expose DRAM bytes, energy, and clock, which draw the roofline. A signpost trace confirms each dispatch wraps three driver requests. With boot security lowered on a wipeable machine, kernel function-boundary tracing captured the expanded program that never exists on disk: a list of 44-byte register writes that wire buffer addresses into DMA engines. The M1 firmware is unencrypted, an ARM64 real-time kernel image, and static analysis recovered a 93-command host protocol. Every substantive claim is marked measured, decompile-derived, or predicted.
The naming rule is compact: an M(n) chip has the H(n+12) architecture, so M1 is H13 and M5 is H17. One compiler binary covers 28 targets from A11 through A18 and M1 through M5; only a per-target data table changes. A capability bit is not a reachable operation. Three-dimensional convolution is advertised and recognized by the frontend, then fails backend lowering on every device mask. Operations marked native in Appendix A were compiled and run on the M1.
M1 roofline constants:
| Quantity | M1 measured |
| Overhead-isolated matmul slope | 12 fp16 TFLOP/s |
| Saturating large matmul | 4.8 fp16 TFLOP/s |
| Convolution end-to-end | 1.8 TFLOP/s |
| DRAM ceiling / weight stream | 85 GB/s / 51 GB/s |
| Ridge point | 141 FLOP/byte |
| On-chip working set | 2 MB |
| Per-dispatch floor | 0.23 ms |
A 256-channel 3x3 convolution runs about 3.8 times faster than the same chip's GPU and about 9 times more energy-efficient. A sixteen-deep conv stack hits 2063 GFLOP/s/W on the M1 engine against 142 on the GPU, a 14.5x efficiency edge; on the M5 the pair is 2289 vs 175, about 13x. On the M2 a ViT-B/16 forward costs 67.9 mJ against the GPU's 714.3 mJ, about 10.5x. Idle engine power is about 0 W (rail-off). A sustained 3x3 convolution draws about 1.78 W. Optimum efficiency is near 0.37 pJ/FLOP. A 3.5-minute compute-bound loop holds one clock state.
The datapath multiplies in fp16 end to end; the accumulator is wide, of fp32 class. Apple's marketed 16-core figure is not the throughput unit. The compiler tiles across the HAL offset 0x238 count: 4 cores on the M1 base, 8 on Pro/Max, 16 on the M5. Each core emits 4 output channels per cycle on the default fp16 path. On the unentitled path, int4 lookup-table weights run about 2.37 times faster than fp16; structured sparsity is 1.55 to 1.64 times faster at 0.43 times the bytes. On the M1, int8 affine folds to dense fp16 in DRAM, so it saves stored size and not bandwidth.
Autoregressive decode is both bandwidth-bound and dispatch-bound. One transformer layer stack issues on the order of 40 to 50 small dispatches, each paying the 0.23 ms floor. At batch 16 the GPU is about 2.7 times faster and about 4.6 times more energy-efficient on decode. Native int8 weights halve traffic yet leave a hybrid decoder at about 0.99 times the fp16 rate, because wall time is set by dispatch floors. Send prefill, encoder, and the vision front end to the engine; send decode to the GPU. The down-projection over a contraction of 5632 takes about 3 percent fp16 error, enough to flip greedy argmax.
There is no backward operation. Training still runs: the host builds a vector-Jacobian-product graph from ordinary forward ops. The registered gradient set matches closed form to a cosine of 1.0000. A small conv net reaches 0.9080 test accuracy on the M1 and 0.9070 on the M5 after 300 seeded steps, a one-sample gap. That is a capability claim, not a speed claim. At this scale the loop is dispatch-bound.
This is the most complete public account of the ANE from the fp16 array down to the firmware protocol. For on-device vision, encoders, short-sequence attention, and mid-size matmul, the engine is both faster and cheaper in energy. Large square GEMM, long-sequence attention, and autoregressive decode belong on the GPU. Tiny ops below the 0.23 ms floor belong on the CPU. In serving, a true-batched encoder block crosses to the GPU near batch 23 and self-attention near batch 6; vision convolution never crosses from batch 1 to 256.
For anyone squeezing on-device latency and battery, the first knobs are the 2 MB working-set cap, fusing the graph into one dispatch, and picking a weight format that actually streams on the target. The M5 raises the working set to 4.72 MB, the matmul slope to about 19.6 TFLOP/s, and the weight stream to about 145 GB/s. The programming model does not change.
The direct path is not for shipping software. Scatter, reduceprod, and the GRU/LSTM/RNN cells have no hardware path on any family. Symbolic shapes do not lower on the unentitled path, so each concrete shape is a separate compile. Dynamic-weight convolution at batch 2 or more crashes the compile service. On M1 and M2, a last-axis slice with a nonzero start offset saturates to infinity above 4094.
Power figures come from powermetrics, a modeled estimate, with no independent wall-meter check. Direct silicon is the M1 and M5, plus a partial M2 pass; M3 and M4 rows are predicted from device tables. Private symbols are tied to ANECompiler 9.509.0; a macOS update can break the path. Training at the demonstrated scale is no faster than the host or the GPU.