GPU Kernels Explained: The Hidden Reason Your AI Model Runs Slowly

Paolo Perrone

11 min read

·

Oct 7, 2026

--

6

Listen

Share

A GPU kernel is the code that runs on the GPU when you call an operation like torch.matmul. Instead of running it once, the GPU runs thousands of copies, each on a different piece of the result. Most operations have several kernels, each tuned for different tensor shapes. PyTorch’s math library picks one for your shapes, and that pick sets how fast your model runs.

You are inside the ⚡ Hardware & Inference Course Every lesson free, in order.

TL;DR

• Kernels, the small programs a GPU runs for each PyTorch call, set your model’s speed.
• Which kernel runs depends on the sizes you choose: adding 47 unused entries to nanoGPT’s vocabulary made its training 25 percent faster.
• What also slows kernels is waiting on memory: in BERT, a Google model, 0.2 percent of the math took 37 percent of the time.
• Fixing that means merging operations into one kernel, which runs two cosine operations about twice as fast.
• 🔒 Paid subscribers get kernel_speedups.ipynb and kernel_trace_check.py. The notebook shows three speedups on your own GPU in two minutes, up to 8.5x on a free Colab T4. The checker reads a timeline of your training steps and names the one change your step time needs. Upgrade to get kernel_speedups.ipynb and kernel_trace_check.py.

📬 New here? Every week, one AI engineering concept made usable…