gpu

A GPU trades a CPU's small number of complex, independent cores for thousands of much simpler cores built to run the same instruction across massive amounts of data at once — NVIDIA's H100, the chip behind most large-scale AI training as of this writing, packs 16,896 CUDA cores alongside 80 gigabytes of HBM3 memory delivering roughly 3.35 terabytes per second of bandwidth, compared to a high-end server CPU's few dozen cores and a small fraction of that memory bandwidth. NVIDIA's CUDA platform, introduced in 2007, is what turned this parallel hardware from a graphics-only tool into a general-purpose computing platform researchers could actually program, and it remains the dominant software layer underneath the vast majority of deep learning training and inference today.

That lockstep execution is exactly what makes a conditional branch inside a warp expensive, in a way that has no real equivalent on a CPU. If the 32 threads in a warp hit an if-statement and some threads take the "if" branch while others take the "else" branch, the hardware doesn't run them truly in parallel — it executes the "if" path for the whole warp while masking off the threads that should have taken "else," then executes the "else" path while masking off the rest, meaning a single branch that splits a warp roughly in half costs close to double the instructions for that warp compared to a branch everyone agrees on. This is a big part of why GPU-friendly code avoids data-dependent branching wherever possible, and why algorithms that process every element the same way — exactly what matrix multiplication and convolution do — map onto this hardware so naturally.

Tensor Cores, specialized hardware NVIDIA introduced with its 2017 Volta architecture, are a second, separate bet layered on top of the general-purpose CUDA cores, purpose-built for the one operation deep learning runs constantly: matrix multiplication. Rather than running matrix multiply through general-purpose arithmetic units the way the CUDA cores do, Tensor Cores perform small matrix-multiply-and-accumulate operations in dedicated hardware, delivering dramatically higher throughput for exactly this one operation than the same chip's general-purpose cores could reach — which matters enormously for deep learning specifically because both the attention mechanism inside transformers and the convolutions inside vision models reduce, at the arithmetic level, almost entirely to large matrix multiplications.

Memory bandwidth, not raw core count, is the bottleneck for a large share of real AI workloads, which is why HBM capacity and bandwidth numbers get as much attention as core counts in GPU announcements. A GPU with an enormous number of cores still sits idle waiting on data if it can't be fed fast enough, and large language model inference in particular is often memory-bandwidth-bound rather than compute-bound — the model has to read its entire set of weights from memory for every token generated, so a GPU's HBM bandwidth, not its peak FLOPs, frequently sets the real ceiling on how fast it can actually serve a large model, which is exactly why each generation of AI-focused GPU has pushed HBM bandwidth as hard as it has pushed core count.

A GPU's entire design is a bet that most of the work worth doing at scale looks the same across millions of data points — the same pixel shader across every pixel, the same matrix multiply across every attention head — and neural networks turned out to be almost the ideal case for that bet, which is the specific historical accident that made a chip originally built to render video game graphics the hardware modern AI training runs on instead.