You run kernels, not models: why the same model and GPU can perform wildly differently

Roger_M_Taylor · x · 2026-09-23

Ahmad Osman lays out why inference performance lives below the model layer: what you actually run are kernels, not models.

A model is just a graph; the inference engine is scheduler/optimizer/executor, but the real work happens in kernels — MatMul, attention, RMSNorm, KV cache, quantized linear, sampling, and fused kernels that avoid writing back to memory repeatedly
Same model, same GPU, same VRAM can perform wildly differently: one stack uses hardware-aware fused kernels, the other shuttles tensors through 47 tiny launches and blames the GPU
His earlier point: don't pick an inference engine first — pick a hardware strategy, workload shape, and serving model; below that, pick a file encoding and kernel path
Analogy: the model is the recipe, hardware is the kitchen, kernels are the knives and pans

More from Infra