2026-08-13
The paper turns deep-learning architectures into one formal object by making broadcasting a tracked algebraic structure, enough to encode all of DeepSeek-V3.
Deep-learning models are precisely defined mathematical functions, yet the way we describe architectures is strikingly informal: a mix of tensor notation, framework-specific code, and hand-drawn diagrams. These representations work locally, but none is a single formal object you can reason about systematically. As a result, things that should be derivable from an architecture's definition, such as equivalent reformulations, efficient low-level implementations, performance models, and cross-framework realizations, are found by manual derivation and engineering intuition rather than by a mechanical procedure.
The authors pin the problem on a neglected concept: broadcasting. SoftMax written as a map from R^x to R^x can mean a row-wise operation or an operation over a whole tensor, depending on which axis is "tiled" and which is the target. Strip the broadcasting information and the expression becomes mathematically ambiguous. Broadcasting is also exactly what maps an architecture to parallel hardware: GPU kernels are defined against broadcast axes, and bandwidth-saving tricks like FlashAttention live or die on stating broadcasting correctly.
The tool is category theory, the mathematics of composition and abstraction. The authors build a layered category for deep-learning architectures whose central move is to promote broadcasting from implicit convention to a tracked, first-class structure. The machinery is two new categories: one for affine stride transformations over indices (axis-stride), and an array-broadcasted category Br, written [B;A], where B is the base datatype and A the array axes.
The key device is the weave. A weave tags each index axis as "tiled" (broadcast over) or "target" (what the operation actually acts on), and welds this information into the operation itself rather than leaving it outside. The resulting diagrams are both compact and computationally meaningful. On this base the authors reassemble the building blocks of deep learning: Einstein operations such as transposes, sums, outer and inner products, using reindexings plus a dashed wire for multiplication and a curved cup for contraction, recreating Penrose graphical notation; SoftMax as a polymorphic operator whose tiling axis the weave decides; Dropout as a probabilistic Markov kernel, with the base category swappable for the probabilistic Stoch; learned layers and norms (RMSNorm) via the Para construction, which hides weights as a side input; embeddings as a selection over an integer datatype; and convolution as an addition reindexing followed by a linear layer, whose translational equivariance can be derived by sliding the index through.
One detail shows why the formalism pays: Dropout is a non-deterministic random operation, so copying an axis of random values is not the same as running the random operation in parallel across those axes. The algebra reflects this (Yoneda sliding fails on non-deterministic operations); informal notation would paper over the difference.
This is an infrastructure paper; the results are capability demonstrations, not benchmarks. The authors encode the full DeepSeek-V3 architecture: the standard transformer trunk, multi-head latent attention with a complex rotary and a latent real component, GeGLU feed-forward blocks, and a mixture-of-experts (MoE) layer with a gate. Around this expression they ship:
| Capability | What it does |
| Dual-language packages | pyncd (Python) and tsncd (TypeScript), mirrored and JSON-interopable |
| Autoalignment | Axes auto-matched on composition, e.g. qkmatmul @ softmax @ mask @ svmatmul; framed as a structure-preserving functor |
| Configuration generation | Free axes remaining after alignment are the model's degrees of freedom, scanned into config items |
| PyTorch compilation | An algebraic expression compiles to a runnable PyTorch module; PyTorch is incidental, TensorFlow or Triton would work too |
| Hypergraph rewriting | Converting to a hypergraph enables algebraic rules: associativity, bifunctoriality, symmetry |
| Diagram generation | The TypeScript renderer produced the architecture figures in the paper |
There are no accuracy or speedup numbers. The paper does not claim models get faster or more accurate; the deliverable is that a production-grade SOTA model can be fully expressed, and that the expression is machine-manipulable.
The audience is not model trainers but the people building compilers, kernels, and frameworks, and those doing architecture-hardware co-design. The tell is in prior work: the authors' 2025 "FlashAttention on a Napkin" showed that once broadcasting is stated explicitly, FlashAttention and its performance model can be derived procedurally. This paper solidifies the mathematical foundation so such derivation can be automated and extended to arbitrary architectures, instead of being redone by hand for every attention variant or new chip.
The longer-horizon payoffs include turning backpropagation itself into an algebraic operation via Para (an alternative to PyTorch's autograd) and analyzing how quantization error composes, relevant because bandwidth cost is superlinear in quantization size and bandwidth is exactly what current models are bottlenecked on.
The authors themselves defer the most consequential capability to future work: automatically deriving low-level kernels (tiled matmul, attention) and hardware-aware performance models. None of that is done here; only the foundation is laid. Para-style algebraic autodiff and quantization-error composition are proposals, not implementations.
The MoE "sparse axis" form is mathematically coherent but, as the authors admit, insufficient for direct computation; it must be algebraically rewritten into an executable form before it runs. PyTorch compilation needs extra infrastructure for PyTorch's elaborate broadcasting semantics. The largest open question is that every engineering payoff currently depends on downstream tools being built on this foundation; the foundation itself has no benchmark showing a quantifiable edge over existing notation like einops or named tensors.