Why Isn't On-Device AI Improving With Better Phone NPUs?

The Limitations of On-Device AI: Why Smartphones Still Lag Behind Cloud-Based Models

Smartphone manufacturers are constantly pushing the boundaries of artificial intelligence, touting ever-faster neural chips and promising groundbreaking features. Yet, when it comes to actual user experience, the AI capabilities on these devices often feel underwhelming compared to the powerful chatbots and image generators available in the cloud. This gap isn't just a matter of marketing hype; it reflects a fundamental mismatch between what today’s neural processing units (NPUs) are built for and what modern AI models actually require.

To understand this disparity, we need to look at the interplay of money, physics, and software that lies between the flashy "AI phone" presentations and the apps people use daily.

NPUs Are Racing Ahead, But TOPS Is a Blunt Instrument

On paper, the progress in NPU performance is impressive. Every new flagship device arrives with a higher number, as if more specifications could finally bring sci-fi-level AI to your lock screen. The standard metric used to measure this progress is TOPS, or "trillions of operations per second," which has become the shorthand for how fast an NPU can perform matrix math. However, chipmakers themselves admit that judging an NPU solely by TOPS is a reductive approach, as it ignores memory constraints, software stacks, and the real-world complexity of applications.

This disconnect helps explain why a phone with triple-digit TOPS can still struggle to run a modern language model smoothly. What matters is not just raw computational power but also how efficiently the chip moves data, schedules workloads, and shares tasks with the CPU and GPU that sit alongside it. As former Apple engineers have pointed out, phones like the Google Pixel and Apple iPhone contain a CPU, GPU, and NPU, but each vendor integrates these components differently. Until software can fully leverage this trio, the headline TOPS figure will continue to overpromise on what on-device AI actually delivers.

Big Models, Tiny Phones: The Physics Problem

The other half of the story is that the AI people expect on their phones is designed for an entirely different environment. The most advanced systems—such as large language models and diffusion image generators—are trained and fine-tuned on massive parameter counts that demand significant compute and memory for both training and inference. According to computational linguist Graça Nunes, while almost all NLP and AI problems can be solved with LLMs, these systems rely on powerful servers and substantial electricity.

Phones, on the other hand, operate under strict power and thermal limits. Pushing the silicon too hard leads to overheating and rapid battery drain. Without robust memory and storage, even simple AI models can be constrained by latency, power, and bandwidth. One chipmaker bluntly calls memory a strategic enabler of mobile AI. The physics of heat, battery life, and memory mean that no matter how fast the NPU becomes, a phone will never behave like a rack of GPUs in a data center.

Software Is Lagging the Silicon

Even where the hardware is ready, the software often isn’t. Adapting current neural networks to take full advantage of NPU capabilities requires specialized knowledge and development effort. Adapting models to NPU architectures is still a niche skill. On iOS, Apple has attempted to smooth this path with Core ML, its flagship framework for machine learning, which includes tools to convert and optimize models for Apple devices. However, even with Core ML, developers must redesign networks, prune parameters, and balance precision to meet mobile constraints.

Across platforms, the most successful AI infrastructure relies on specialized accelerators like TPUs or GPUs and highly optimized frameworks such as JAX, TensorFlow, or PyTorch that maximize hardware utilization. Phones are only beginning to get this level of tooling, and the fragmentation between Android vendors and iOS makes it harder for app developers to create a single well-tuned on-device model. Until the software ecosystem catches up, the NPU inside your phone may often idle while the CPU or GPU handles the workload.

Compression, Quantization, and the Quality Trade-Off

To fit big models into small devices, engineers use compression techniques that come with real trade-offs. Quantization, which reduces the precision of weights and activations, is one of the most important, but research shows its limitations. While quantization may degrade performance compared to full-precision models as precision decreases, new methods are trying to learn better step sizes. However, preserving accuracy under aggressive compression remains a challenge.

That is why some on-device assistants feel a generation behind their cloud counterparts—they are, in a very literal sense, smaller and blurrier versions of the same models. On-device processing must operate within the power and thermal limits of mobile devices, restricting the size and complexity of models compared to AI systems running on dedicated server farms. The result is a constant balancing act between responsiveness, battery life, and output quality, and for now, many phone makers are choosing to keep models small enough that they rarely wow users in the way cloud tools can.

Why On-Device AI Still Matters

Despite these limitations, the industry is not chasing on-device AI as a vanity project. Running models locally can be a game-changer for latency and reliability, which is why Samsung’s chip division argues that the integration of on-device AI is truly a game-changer for mobile technology, minimizing lag and enhancing user experiences while unlocking the full potential of on-device AI.

That matters for features like camera processing, live translation, and voice commands, where even a half-second round trip to the cloud can make the experience feel sluggish. Privacy is another major driver. Better privacy and security are central selling points for AI PCs, where a powerful NPU lets AI processing happen locally so sensitive data never leaves the device, ensuring that personal information stays under the user’s control. The same logic applies to phones. Privacy and security benefits are strongest when data is processed locally, addressing growing privacy concerns and regulatory pressure.

Edge AI Is Rising, But the Cloud Still Dominates

For years, AI discourse has been dominated by massive cloud-based models trained on enormous datasets and running in centralized data centers. That gravitational pull has shaped what users expect from “AI.” Even as companies talk up edge computing, the biggest breakthroughs still arrive as cloud services that phones tap into over a network. Analysts describe how AI discourse has been dominated by this centralized model, even as intelligence starts to spread to the edge.

At the same time, the economics of cloud AI are daunting. Training the most advanced models is extraordinarily expensive and time-consuming, costing tens or hundreds of millions of dollars. Training the largest systems makes jurisdiction-specific variants economically infeasible. Despite this popularity, training and deploying these models in the cloud can still be challenging, and aligning them with business objectives is time-consuming and compute-intensive. That cost pressure is one reason vendors are eager to offload some inference to phones, even if the user experience is not yet on par with the cloud.

Developers Are Still Figuring Out How to Use NPUs

From an app developer's perspective, on-device AI is not a free upgrade—it is a design problem. On-device AI processing keeps everything local, which is faster and works offline, but a phone’s processor needs to handle all the heavy lifting. Developers are reminded of this when they try to bolt a chatbot onto a messaging app. Furthermore, these endeavors to enhance optimization have proven to be time-consuming and labor-intensive, often requiring several iterations for design and deployment.

That complexity is compounded by the fact that the NPU is designed to provide a base level of AI support extremely efficiently, but is generally limited to small, persistent solutions that still need to have long battery life. That makes NPUs perfect for always-on tasks like wake-word detection or background photo enhancement, but less suited to the bursty, heavyweight workloads of a full generative model. Until developer tools make it easier to split work intelligently between CPU, GPU, and NPU, many apps will stick to simpler, more predictable uses of on-device AI.

Local AI Is Improving, Just Not Evenly

Despite the frustrations, there are signs that on-device intelligence is quietly getting better. Artificial intelligence is moving toward an on-device platform from the cloud platform for better reliability, more privacy, and lower latency. Researchers have already built pipelines like Fontnet that can run on a resource-constrained mobile platform. In practice, that shift shows up in features like offline transcription, on-device spam filtering, and camera modes that no longer need a network connection to work.

The broader local AI ecosystem is also maturing. Analysts argue that the future of local AI is bright, driven by advancements in dedicated hardware and privacy-preserving technologies that make it a fundamental component of modern digital products. The local AI landscape is evolving quickly, and while today’s tools may feel a bit raw, experts expect them to be dramatically better in just a year. That uneven pace helps explain the user experience: some features, like photo processing on a 2025 flagship, feel magical, while others, like on-device chat, still lag.

Hybrid AI Is the Realistic Endgame

Given all these constraints, the most plausible future is not a phone that replaces the cloud, but a partnership between the two. The most likely outcome? A hybrid future where AI and human models coexist, with AI systems dominating certain sectors while humans retain a stronghold in areas requiring authenticity and relatability. The same hybrid logic is emerging in infrastructure: with the advanced capabilities of on-device AI, a hybrid AI architecture can scale to meet enterprise and consumer needs, providing better cost, energy, performance, privacy, security, and personalization than either side alone.

Phone platforms are already moving in that direction. Apple’s latest strategy gives developers free access to on-device large language models, and analysts expect that move will accelerate the trend of hybrid AI architectures, where on-device models handle common tasks and cloud models are reserved for more complex or data-intensive queries. In parallel, PC makers explain that what on-device AI means in relation to NPUs is that machine learning tasks run locally on a device’s NPU instead of in the cloud, handling things like processing images or audio directly and instantly. As that pattern spreads to phones, the most satisfying AI experiences are likely to be the ones users barely notice, where the NPU quietly handles the routine and the cloud steps in only when it truly adds something new.

The Hype Cycle Is Ahead of the User Experience

There is also a timing problem. Investors and marketers sprinted ahead of the technology, promising a revolution before the application layer was ready. Analysts now argue that, as with most new technologies, it takes time for generative AI to mature and reach application-layer development and distribution. On phones, that lag shows up as a mismatch between the bold “AI phone” branding and the relatively incremental features that ship at launch.

Some critics worry that the imbalance of power in AI will only deepen as the biggest players control both the cloud models and the chips in people’s pockets. Author Gary Rivlin has warned that creating these models concentrates so much power in the hands of a few, a concern he voiced when discussing creating large AI systems and deregulation. For now, though, the more immediate frustration for users is simpler: phone NPUs really are getting better, but until the physics, software, and business incentives line up, the AI they enable will keep feeling like a work in progress rather than the leap the marketing promises.