On-Device Runtimes and Neural Processing Units

For years, many artificial intelligence features worked like a long-distance call: an app sent data to a cloud service, the service ran a model, and the answer came back over the network. On-device inference changes the shape of that system. The model runs inside the phone, laptop, camera, robot, or factory gateway itself. That sounds simple, but it depends on two quiet pieces of engineering: a runtime that knows how to execute the model, and hardware such as a neural processing unit (NPU) that can do the math quickly while using little power.

Start with the job: turning inputs into predictions

A model is a program that has learned patterns from data. At run time, it takes an input, performs many numeric operations, and produces an output. The input might be a sentence, an image, a sound clip, a sensor reading, or a mix of all of them. The output might be a label, a summary, a translation, a detected object, or a recommended next action.

Running the model to produce an output is called inference. Training creates or improves the model; inference uses it. On-device inference means this use happens locally on the device rather than in a remote data center. The distinction matters because local execution changes latency, privacy, cost, reliability, power use, and the way software must be shipped.

📌

Inference is use. Training teaches a model; inference is what happens when an application asks the trained model for an answer.

What a runtime actually does

A runtime is the layer of software that makes a model executable on real hardware. The model usually arrives as a file that describes operations such as matrix multiplication, convolution, normalization, attention, or activation functions. The runtime reads that description, checks what the device can do, chooses an execution plan, moves data through memory, and calls the right hardware instructions.

Without a runtime, every app would need to know the details of every processor, driver, memory layout, and model format it might encounter. The runtime hides much of that complexity. It can send some work to a central processing unit (CPU), some to a graphics processing unit (GPU), and some to an NPU, depending on speed, power, supported operations, and available memory.

A common example is ONNX Runtime, where ONNX stands for Open Neural Network Exchange. It lets a model be represented in a portable format and executed through different hardware back ends. The important principle is broader than any one tool: the runtime is the contract between the model and the machine.

A runtime reads the model, plans execution, and dispatches operations to the right hardware.

Why NPUs exist

A CPU is a general-purpose processor. It is excellent at branching logic, operating system work, and many small tasks that change shape constantly. A GPU is built for doing many similar operations in parallel, which made it valuable for graphics and later for model training and inference. An NPU is more specialized still: it is built for the repeated tensor math that many modern models require.

A tensor is a block of numbers arranged in one or more dimensions. A sentence can become a sequence of number vectors; an image can become a grid of number channels. Models spend much of their time multiplying and adding these numbers. An NPU is designed to do that pattern efficiently, often with lower power use than a CPU or GPU for the same model.

Specialization is not magic. An NPU is fastest when the model uses operations it supports, when data fits well in memory, and when the runtime can avoid expensive copying between processors. If the model contains unsupported operations, the runtime may fall back to the CPU or GPU for those parts. That fallback can be useful, but it can also reduce the speed and power benefits.

Use the CPU for flexible control logic and operations that do not benefit from heavy parallelism.
Use the GPU when broad parallel compute is available and power or sharing with graphics workloads is acceptable.
Use the NPU when the model fits its supported operations and low power inference is a priority.
Measure the whole pipeline, not just the processor, because memory movement can dominate small workloads.

CPUs handle any task, GPUs run parallel math, and NPUs specialize in efficient tensor operations.

The hidden work around the model

A model is only part of an on-device feature. Before inference, the app usually prepares the input: resizing an image, converting audio into features, tokenizing text, normalizing sensor readings, or combining signals. After inference, the app interprets the output: filtering detections, ranking candidates, formatting text, or applying business rules.

These steps matter because they may run on different processors from the model itself. A vision model that runs quickly on an NPU can still feel slow if image decoding, resizing, or post-processing stays on the CPU and copies large buffers through memory. Good on-device design treats the feature as a pipeline, not a single model call.

The runtime helps, but it cannot remove every constraint. Device memory is finite. Battery and heat budgets are real. Multiple apps and system services may compete for compute. A well-designed feature uses the NPU where it helps, keeps data movement small, and has a graceful path when the ideal accelerator is unavailable.

⚠️

Watch the copies. Moving data between processors can erase the gains from a fast accelerator.

An on-device feature is a pipeline: prepare the input, run inference, then interpret the output.

Making models small enough to live on the device

Many models are too large or too expensive to run comfortably on a personal device in their original form. Engineers reduce the cost through several techniques. Quantization stores numbers with fewer bits, such as using 8-bit integers instead of 32-bit floating-point values. Pruning removes parts of the model that contribute little. Distillation trains a smaller model to imitate a larger one.

These techniques involve trade-offs. A smaller model may be faster, cheaper, and cooler, but it may also be less accurate or less capable on unusual inputs. The right question is not whether the smallest model wins. The right question is whether the model is good enough for the product requirement under the device constraints.

The model format and runtime also need to match the target devices. A model that performs well on one NPU may need conversion, operator selection, or calibration before it performs well on another. Teams often keep a reference model for quality testing and produce optimized variants for specific device classes.

Quantize when power, memory, or latency matter more than small changes in numeric precision.
Prune when the model has redundant structure that can be removed without harming required quality.
Distill when a large model teaches a smaller model to handle the product's common cases.
Keep test data that represents real users, real sensors, and uncomfortable edge cases.

Cloud, device, or both

On-device inference is not the opposite of cloud inference. It is another placement choice. The cloud is useful when the model is very large, the device is weak, answers can tolerate network delay, centralized updates are important, or the task needs data that lives in a service. The device is useful when response time must be immediate, connectivity is unreliable, data should stay local, or per-request cloud cost would be too high.

Many strong designs use both. A device can run a small local model for instant response and privacy-preserving first pass decisions, then call a cloud model for harder cases, richer reasoning, or background improvement when the user permits it. A model selected or evaluated in Microsoft Foundry can still be packaged for local execution if the runtime, license, and hardware target support that path.

The placement decision should be made per feature, not per organization. A keyboard suggestion, wake word, camera effect, or anomaly detector may belong on the device. A long document analysis job or large multimodal assistant may belong partly or mostly in the cloud. The boundary can change over time as hardware, runtimes, and model efficiency improve.

Immediate, offline, privacy-sensitive work runs on the device; large or centralized work runs in the cloud.

Decisions leaders and practitioners must make

The practical question is not whether an NPU is good. The question is whether local inference improves a specific outcome enough to justify the engineering and operating work. A leader may care about user experience, cloud spending, privacy posture, and product differentiation. A practitioner may care about model format, supported operators, memory pressure, thermal behavior, testing, and update paths. Both views are necessary.

Start with the service level you need: maximum acceptable latency, offline behavior, battery impact, quality threshold, and privacy requirement. Then test representative devices, not only developer machines. Measure cold start time, steady-state latency, power draw, memory use, accuracy, and fallback behavior. If a feature only works on a narrow set of devices, the product needs a clear compatibility story.

Governance also changes when models ship to devices. Updates may be slower than cloud updates. Users may stay on old app versions. Model files can be inspected or copied more easily than server-side models. Sensitive data may be safer because it stays local, but sensitive logic may be more exposed. These are design issues, not reasons to avoid on-device inference.

✅

Measure first. The best accelerator is the one that improves the full user experience on the devices your users actually have.

Key takeaways

On-device inference runs a trained model locally, reducing dependence on a network round trip.
A runtime turns a model file into executable work across the CPU, GPU, NPU, drivers, and memory system.
An NPU is specialized for tensor math and can improve latency and power use when the model fits its capabilities.
The whole feature pipeline matters; preprocessing, post-processing, and data movement can dominate performance.
Smaller models usually require trade-offs among speed, memory, power, accuracy, and device coverage.
Cloud and device inference are placement choices that can be combined within one product.

Glossary

Inference: Inference is the act of using a trained model to produce an output from a new input.
Runtime: A runtime is software that loads a model, plans its execution, and runs it on available hardware.
Neural processing unit: A neural processing unit is a processor designed to run common artificial intelligence model operations efficiently.
Central processing unit: A central processing unit is a general-purpose processor that handles operating system work, application logic, and many varied tasks.
Graphics processing unit: A graphics processing unit is a processor designed for highly parallel work, originally for graphics and now also used for model computation.
Tensor: A tensor is a structured block of numbers, such as a vector, grid, or higher-dimensional array used by models.
Quantization: Quantization is the process of representing model numbers with fewer bits to reduce memory use and computation cost.
Fallback: Fallback is the path a runtime or application uses when the preferred hardware or operation is not available.