Running AI on Your Own Device: The Edge Reality Check

The pitch for local AI is simple and mostly true: privacy, no API bill, works on a plane. The pitch also quietly skips the part where your phone slows down, your laptop battery halves, and the hardware you own turns out to matter more than the model you downloaded.

Researchers benchmarked the same model — Qwen 2.5 1.5B, 4-bit quantised — on four very different machines, using an identical 258-token prompt, greedy decoding, and 20 consecutive iterations per device (arXiv:2603.23640, June 2026).

The hardware

The results

Three conclusions fall straight out of that table.

First, thermal management beats peak compute on mobile. The iPhone started above 40 tokens per second, then the iOS thermal governor moved it from Normal to Warm and throughput dropped 38.4% within two iterations. Not twenty. Two.

Second, the S24 Ultra did something worse than slow down. It hit a hard OS-enforced GPU frequency floor that terminated inference entirely. This is the failure mode nobody budgets for: not degraded output, no output.

Third, the slow chip won on energy. The Hailo-10H managed 6.9 tok/s at under 2 watts — 19× lower throughput than the laptop GPU, but comparable energy proportionality. For an always-on assistant that sits idle most of the day, that is the trade you actually want.

Why this matters beyond benchmarks

The devices are getting cheaper fast. A companion survey (arXiv:2604.24785) lists accelerators like the Hailo-10H at 40 TOPS, the Jetson Orin Nano Super at 67 TOPS, and integrated NPUs bringing usable AI compute to hardware under $350.

The same survey reports that INT4 quantisation cuts model size 4× to 16× relative to FP16 — a 75% to 93.75% reduction — with hardware evaluations showing 57–61% reductions in area and power consumption.

So the pipeline works. But the efficiency ranking is not what hardware marketing suggests. The M5Stack and Jetson GPU landed at 0.5–1.7 MJ/Mtok, the most efficient of the group. The Raspberry Pi with its NPU hat ranged from 0.8–5.5 MJ/Mtok, varying far more by model. That suggests NPU efficiency is more sensitive to model architecture than GPU inference is — a detail that will bite anyone who assumes an NPU is interchangeable.

Specialised silicon pushes further still. EdgeCIM (arXiv:2604.11512) reports up to 7.3× higher throughput and 49.56× better energy efficiency than an NVIDIA Orin Nano on LLaMA3.2-1B, and 9.95× higher throughput than a Qualcomm SA8255P on LLaMA3.2-3B. Moving from INT8 to INT4 roughly doubles throughput and efficiency again. Under INT4, Qwen2.5-0.5B clears 1000 tok/s at over 600 tokens/J.

None of that is on your phone today. It is what the next two years look like.

How to pick hardware without a lab

1. Decide between burst and always-on. Burst throughput matters for interactive work. Efficiency per watt matters for background agents. They reward opposite hardware.
2. Assume sustained load, not peak. Every mobile number you have read is a first-iteration number.
3. Check the thermal envelope, not the TOPS. A device that terminates inference is worth nothing.
4. Watch memory bandwidth, not just compute. Modern decoding is bandwidth-bound; the RTX 4050 hit its battery limit, the Hailo-10H hit its on-module memory ceiling.
5. Match precision to device. INT4 on mobile, higher precision on server-class hardware.

The honest summary

Local AI is real, and it is getting good. But “it runs on-device” is the beginning of a sentence, not the end. The model is the least interesting variable in the system. Your phone’s thermal governor is the one that decides whether you get a working assistant or a warm rectangle that stops answering.

Sources: LLM Inference at the Edge: Mobile, NPU, and GPU (arXiv:2603.23640, June 2026) · Cloud to Edge: Benchmarking LLM Inference on Single-Board Computers (arXiv:2604.24785) · EdgeCIM (arXiv:2604.11512, April 2026) · Google AI Edge Portal, May 2026