Local AI Is a Routing Problem, Not a Privacy Feature
I’ve had a longstanding interest in local AI for a simple reason. I would rather not send proprietary code, unpublished plans, or knowledge about how a business operates to an outside model provider when I have a viable alternative. I want more control over where that information is processed.
A company does not need a model on every employee’s laptop to achieve that. A properly secured internal server can keep information away from outside providers. I admit there’s a bit of wishcasting here. But if my friends are any guide, I’m far from the only one who wants a capable model living on my own laptop.
Learning something each week? Consider becoming a paid supporter 🙏
What has changed is that more tasks can now run on individual devices. As local models improve, I’ve started thinking more seriously about what belongs on a laptop or phone, what needs an internal server, and what still calls for an outside provider. Privacy drew me to local AI. I think the more consequential question is where each task belongs.
The Device Becomes a Place to Run AI
Local AI is already appearing in consumer products. Supported Pixel phones can summarize recordings on the device, while Windows can generate live captions without an internet connection. People use these features without downloading a model or running an inference server. That is part of what makes them useful. Developers can build similar features with open-source projects such as Moonshine Voice, which supports on-device transcription and voice interaction.
I expect more business applications to follow. Tools for running local models can select suitable hardware, download models, and provide interfaces similar to hosted AI services. Developers no longer have to assemble every component themselves.
The hardware is improving too, but I would be careful about generalizing. Microsoft’s Project Zenith targets developer-class PCs with at least 64 GB of memory and 250 GB/s of memory bandwidth. That tells us something about where powerful laptops are heading. It does not mean the typical company laptop can run the same workloads.
There is also a cost to using computers people already own. Models compete with employees’ applications for memory and processing power. Laptops sleep and disconnect. A fleet is worth considering as AI capacity, but it is not a free or continuously available computing cluster.
Which Work Goes Where?
I would start with well-defined tasks whose results are easy to check. Transcription, document classification, local search, and focused coding assistance are candidates for laptops. Shared business data or demanding, always-available applications may call for an internal server. Outside providers remain an option when information is approved for external processing.
An evaluation of local coding models found that narrow assignments with specific instructions worked better than extensive code exploration. Long conversations were less reliable. Phones have tighter memory and heat constraints, making short, specialized tasks a more practical starting point.
The economics are less obvious than token prices suggest. In one study of coding agents, prompt caching made cloud inference cheaper per token, yet shared on-premises infrastructure had roughly 40% lower modeled total cost. Dedicated hardware cost more than the cached cloud API, and the local configuration required substantially more repair work. The study involved one developer and no employee laptops.
I would measure cost per correctly completed task, including failures, rework, response time, and the effect on the employee’s computer. Owning the hardware does not automatically make inference cheaper.
Responsibility Moves With the Work
Moving inference onto employee devices changes where companies manage safety and security. Models and operating systems provide protections, but those protections can differ depending on where a request runs.
Apple lets developers configure guardrails for its on-device foundation models, while its Private Cloud Compute models have policies developers cannot configure directly. Apple also warns that built-in safeguards can miss harms specific to an application. Where inference runs affects both the available protections and the developer’s control.
Model updates can change responses, and platform safeguards can evolve. Developers need to repeat their evaluations when the underlying system changes. Companies must also decide which models employees may install, keep approved versions current, and withdraw problematic releases. Local inference becomes an endpoint-management issue when employees download models on their own.
My minimum requirement is straightforward. A company should know which model handled a request, where it ran, what data and tools it could access, and what happened if it failed. An app shouldn’t send confidential information to an outside provider just because the local model can’t finish a task.
People running agents on personal computers face similar risks. An OpenClaw agent deleted emails despite instructions to seek approval. I personally would not give an agent unrestricted access to personal files, let alone financial accounts! Sensitive accounts should remain off-limits, and difficult-to-reverse actions should require approval enforced by the software, not merely requested in a prompt.
Where AI Runs Becomes a Design Choice
Apple offers a glimpse of where this is heading. Its latest foundation-model family spans on-device execution and Private Cloud Compute, while its developer framework supports additional local and external models. The demonstrated on-device model has an 8,192-token context window, compared with 32,000 tokens in Private Cloud Compute. These destinations are not interchangeable.
I do not expect laptops and phones to replace internal servers and clusters or outside providers. But as local models improve and the tools for running them become simpler, I expect more routine AI work to move onto devices people already use. The question is where each task belongs and how to make that choice reliable, economical, and secure.
Biology Has a Data Problem, Not a Model Problem
One of my favorite presentations at the recent AI Conference came from the founders of Onyx, who asked when biology will get its ChatGPT moment. Their argument is that compute and models aren’t the bottleneck, and neither is the sheer volume of data. Language models inherited an internet where text, images, and intent were already linked. Biology has huge datasets, but genetic, molecular, and clinical measurements are rarely collected from the same people, and almost never over time. That brought me back to a point I’ve been making lately: useful AI data often has to be built, not just collected. I think there’s a big opportunity in creating these domain-specific datasets. Onyx is pursuing an ambitious example, building a virtual immune system that aims to predict how patients respond to disease and treatment.
AI Is a New Game Mechanic
