edge

Edge AI runs a model directly on the device that generated the data — a phone, a camera, a car — instead of sending that data to a remote data center and waiting for a response. The cloud round trip that edge AI skips is not a small cost: a typical cloud inference request carries somewhere in the range of 100 to 300 milliseconds of network latency alone, before the model has even run, while a model executing locally on dedicated hardware can respond in single-digit milliseconds — a difference that matters enormously for anything time-critical, like a camera deciding whether to trigger a shutter or a car's perception system reacting to an obstacle.

Specialized on-chip hardware is what made running a real neural network inside a phone's power budget practical, not just theoretically possible. Apple's Neural Engine, introduced in the A11 Bionic chip in 2017, was among the first mass-market neural processing units built into a mainstream consumer device, capable of roughly 600 billion operations per second dedicated specifically to machine learning math — hardware a general-purpose CPU could technically run the same operations on, just far too slowly and at far too much power cost to be usable in a phone's thermal envelope. Google's Edge TPU and Qualcomm's Hexagon NPU followed the same logic for their own product lines, and by the early 2020s a dedicated NPU block had become a standard, expected component of phone and laptop chips specifically because on-device AI features — camera processing, voice recognition, predictive text — had become standard product expectations that general-purpose compute alone couldn't deliver within battery and heat limits.

Edge models are deliberately shrunk and re-engineered for their hardware, not simply smaller copies of a cloud model that happened not to fit. Quantization and pruning compress a model's size and computational cost, often at only a modest accuracy cost, specifically to fit within a device's power, memory, and thermal budget — and TinyML pushes this furthest, running compressed models through frameworks like TensorFlow Lite Micro on microcontrollers with only kilobytes of RAM, orders of magnitude smaller than the memory a phone has available, for applications like always-on keyword detection that need to run continuously on a battery for months. Tesla's Full Self-Driving computer is the inverse case at the other extreme — a custom edge AI chip built specifically to run inside the car, reportedly delivering on the order of 144 trillion operations per second per chip, with two redundant chips running in parallel, because the decision latency a self-driving system needs is incompatible with any cloud round trip at all, regardless of network quality.

Edge and cloud AI are routinely combined rather than treated as a binary choice, because the right split depends on which cost matters most for a given request. A phone's camera app might run a small, fast model locally for real-time viewfinder effects while sending a harder, less time-sensitive processing step to the cloud for a larger model to finish — getting edge AI's speed and privacy for the common, latency-sensitive case while still tapping cloud-scale compute when the task genuinely needs it. This hybrid pattern is now the default architecture for most consumer AI products rather than an edge case, precisely because very few products need either pure edge-only or pure cloud-only behavior across every single request they handle.

Read edge AI as a question about where computation happens rather than a different kind of AI: the model architecture and the math are frequently the same family of techniques used in the cloud, just compressed and paired with purpose-built silicon, and that single relocation decision is what buys the latency, privacy, and offline-capability tradeoffs edge AI is chosen for — at the structural cost of having less raw compute available than a data center can offer on demand.