inference

Inference is running a trained model's forward pass to answer a real request — the serving side of machine learning, as opposed to training, which is how the model's weights got set in the first place. Training happens once (or occasionally, for retraining); inference happens on every single user request for as long as the model is deployed, which inverts the usual intuition that the hard, expensive part of AI is building the model rather than running it. At a high level, one inference request moves through a serving system like this:

Inference, not training, dominates total AI compute spend industry-wide, and the reason is purely arithmetic. A model is trained once and then serves potentially millions of requests over its deployment lifetime; each of those requests is its own full forward pass through the model's weights, so total inference compute scales with traffic volume in a way training compute never does. For a widely deployed chatbot or API, the cumulative GPU-hours spent on inference routinely surpass the original training run's GPU-hours within weeks of launch, which is the direct reason chip makers increasingly design and market separate product lines optimized specifically for inference throughput and cost-per-token, rather than treating inference as training's smaller cousin.

Latency and throughput pull inference optimization in two directions at once, and real production systems have to choose a point on that tradeoff deliberately. Latency is how long one request takes to answer — critical for an interactive chatbot where a user is waiting; throughput is how many requests a GPU can process per second in aggregate — critical for batch workloads or high-traffic APIs where total cost-per-token matters more than any single response's speed. Batching multiple requests together increases throughput (more work processed per GPU-second) but can increase individual latency if a request has to wait for a batch to fill; techniques like continuous batching and speculative decoding exist specifically to claw back some of that latency cost without giving up the throughput gains batching provides.

Inference-time optimization techniques exist at every layer of the stack because the economics reward squeezing cost out of a workload that runs millions of times, not just thousands. Quantization (running inference in 8-bit or 4-bit precision instead of 16-bit) cuts memory bandwidth and often speeds up computation with minimal accuracy loss; KV-caching avoids recomputing a transformer's attention keys and values for tokens already generated earlier in the same response; model distillation produces a smaller, faster model trained to mimic a larger one's outputs specifically for deployment, trading a small accuracy loss for meaningfully lower per-request cost. None of these techniques change what the model learned during training — they're entirely about making the already-trained model cheaper and faster to run, which is exactly where the economic pressure actually sits once a model reaches production traffic.

Read inference as the recurring bill that training's one-time cost was always going to be compared against: a model is trained once but answers requests indefinitely, and the entire category of inference-optimized hardware, quantization, caching, and batching techniques exists because that per-request cost, multiplied by real-world traffic volume, is where the actual money in deployed AI gets spent.