What Is Edge AI?

Most machine learning of the last decade ran in data centres. A device captured input, sent it over the network, and a server returned a result. That arrangement is simple, keeps models and hardware centralized, and works well — provided there is connectivity, latency is acceptable, and sending the data somewhere is not itself the problem.

Edge AI is the alternative:

Edge AI is running model inference on or near the device that generates the data, rather than sending that data to a remote server.

The phone recognizing a face to unlock, the camera detecting a defect on a production line, the car identifying a pedestrian — none of these can afford a network round trip, and some cannot afford to transmit the data at all.

This article covers what actually drives the choice, the split between training and inference that most introductions skip, the techniques that make large models fit on small hardware, what it costs in capability, and where the cloud remains the better answer.

The Four Reasons to Move Inference to the Edge

Latency

A network round trip costs tens to hundreds of milliseconds, plus processing. For a recommendation that is fine. For a car deciding whether to brake, an industrial system rejecting a part mid-line, or a wake word that should feel instant, it is not.

Local inference eliminates the round trip entirely. This is frequently the deciding factor, and unlike the others it is a hard physical constraint rather than a preference.

Privacy

If the data never leaves the device, an entire category of risk disappears — interception, breaches of a central store, and questions about what a provider retains.

This matters most for biometrics, health data, and audio or video from private spaces. It is also increasingly a regulatory question rather than a preference, since data-protection regimes treat cross-border transfer and central retention as things requiring justification.

Bandwidth and Cost

A single high-resolution camera produces far more data than is practical to stream continuously, and a deployment with hundreds of them multiplies that. Running detection locally and transmitting only events — or nothing — changes the economics entirely.

The relevant comparison is not model quality but total system cost: local compute purchased once against bandwidth and cloud inference billed continuously.

Availability

A device that must work without connectivity has no choice. Remote industrial sites, vehicles, aircraft, agricultural equipment, and consumer devices expected to function offline all fall here.

Training Stays in the Cloud

A distinction introductory coverage often blurs: edge AI generally means edge inference.

Training is compute-intensive, needs large datasets, and runs on clusters of accelerators. That happens centrally. The trained model is then compressed and deployed to devices, which run it forward to produce predictions.

The partial exception is federated learning, where devices compute model updates locally on their own data and send only those updates — not the data — to be aggregated centrally. This keeps raw data on-device while still improving a shared model, and it is used in production for things like keyboard prediction. It remains a specialized approach rather than the norm.

Making Models Fit

Edge hardware has far less memory, compute, and power budget than a server. Several techniques close the gap, and they are usually combined.

Quantization reduces the numeric precision of model weights — from 32-bit floating point down to 8-bit integers, or lower. Memory drops proportionally and integer arithmetic is faster on most edge silicon. Accuracy degrades, though far less than the precision reduction suggests, which is why quantization is the most widely used technique.

Pruning removes weights contributing little to output, producing a smaller sparse model. Realizing the speedup requires hardware or runtime support for sparsity.

Distillation trains a small "student" model to reproduce a large "teacher" model's behavior. The student can substantially outperform a model of the same size trained conventionally.

Architectures designed for the edge — MobileNet, EfficientNet, and the small end of current language model families — are built for constrained hardware from the start rather than compressed afterward.

Formats and runtimes handle deployment: ONNX Runtime, TensorFlow Lite, Core ML on Apple platforms, and for local language models the GGUF format with llama.cpp.

The Hardware

NPUs (neural processing units) are accelerators specialized for the matrix operations inference requires, at far better performance per watt than a general CPU. They now ship in most current phones and increasingly in laptops — Apple's Neural Engine, Qualcomm's Hexagon, and the NPUs in recent x86 laptop processors.

Edge accelerators such as NVIDIA's Jetson modules and Google's Edge TPU target embedded and industrial deployments, offering more capability than a phone within a constrained power envelope.

Microcontrollers sit at the smallest end, running tiny models — wake-word detection, simple anomaly detection — in kilobytes of memory. This niche is often called TinyML.

Consumer GPUs are what most people use to run local language models, where the binding constraint is usually memory capacity rather than compute.

What It Costs You

Edge AI is a trade, and the trade is capability.

Smaller models are less capable. A model that fits on a phone will not match a large cloud model on complex reasoning. For narrow tasks — detect this defect, recognize this wake word — the gap may be irrelevant. For open-ended tasks it is significant.

Updating is harder. A cloud model updates once. A fleet of deployed devices needs distribution, version management, and a rollback plan, across hardware that may be intermittently connected.

Hardware fragmentation. Different NPUs, different runtimes, different quantization support. A model optimized for one target may need substantial rework for another.

Power and thermal limits. Sustained inference drains batteries and generates heat, and devices throttle. Benchmark numbers measured in short bursts do not reflect continuous operation.

Debugging is harder. Diagnosing behavior on a device in the field is considerably more difficult than inspecting server logs.

When the Cloud Is Still Right

Worth stating plainly, since edge AI is often presented as strictly newer and therefore better.

Use the cloud when the task needs a large model's capability, when models change frequently, when inference is occasional rather than continuous, when devices have reliable connectivity and latency is not critical, or when the workload benefits from batching many requests.

Many production systems are hybrid: a small local model handles the common case and filters, escalating to a cloud model for the harder fraction. This captures most of the latency and bandwidth benefit while retaining access to greater capability when it is genuinely needed.

Conclusion

Edge AI runs inference on the device that produces the data instead of sending it away. The four things that drive that choice are latency, privacy, bandwidth cost, and operating without connectivity — and where none of them applies, the cloud remains a perfectly good answer.

What makes it practical is a set of compression techniques, principally quantization, combined with silicon designed for inference rather than general computation. What it costs is model capability, plus real operational complexity in deployment and updates across heterogeneous hardware.

The framing worth keeping is that this is a placement decision, not a technology tier. Ask where the data is produced, how fast the answer is needed, what it costs to move, and what happens when the network is unavailable. The answers usually make the choice obvious, and quite often the honest answer is a hybrid of both.

Frequently Asked Questions

What is the difference between AI and edge AI?

AI describes the models and techniques themselves; edge AI describes where they run. Edge AI means executing inference on or near the device generating the data — a phone, camera, vehicle, or sensor — rather than sending that data to a remote server. The models are broadly the same kind, usually compressed to fit constrained hardware. It is a deployment choice rather than a different category of technology.

What are some examples of edge AI?

Common ones include face recognition for phone unlock, computational photography, and voice wake-word detection, all of which run on-device. Industrial examples include cameras inspecting products on a production line and predictive maintenance on equipment. Vehicles run perception models locally because a network round trip is unacceptable for safety decisions. Smart home devices detect people or packages without uploading video, and running a local language model on a laptop with a tool such as llama.cpp is edge AI too.

Is edge AI free?

The inference itself has no per-request charge, which is the main economic appeal — you are using hardware you already own rather than paying a provider per call. That is not the same as free. Costs move to capable hardware, engineering time to compress and optimize models for specific targets, and ongoing deployment and update infrastructure across a device fleet. For running a local model on a computer you already have, the marginal cost genuinely is close to zero apart from electricity.

How much does edge AI cost?

There is no single figure because it is not sold as a service with a per-unit price. Costs break down into hardware, which ranges from a few dollars for a microcontroller to hundreds for an embedded accelerator module or consumer GPU; engineering effort to optimize models for the target, which is often the larger expense; and fleet management for deployment and updates. The relevant comparison is this total against continuing cloud inference and bandwidth charges, which favours the edge as request volume grows.

Does edge AI need an internet connection?

Not for inference, which is one of its main advantages — a device with a local model works offline. Connectivity is typically still wanted for deploying model updates, sending results or telemetry, and monitoring. Systems designed for genuinely disconnected operation handle updates through periodic connections or physical media. The key point is that the decision the model makes does not depend on the network being available at that moment.

What is an NPU and do I need one?

An NPU, or neural processing unit, is a processor specialized for the matrix operations neural network inference requires, delivering far better performance per watt than a general-purpose CPU. Most current phones include one, as do many recent laptops. You do not strictly need one — models run on CPUs and GPUs — but an NPU makes sustained inference practical within a battery and thermal budget. For running larger local language models, GPU memory capacity is usually the more important specification.