Local + Cloud
When Local Isn’t Enough
By the end of this module, you’ll have a clear framework for deciding when to run locally, when to use an API, and how to wire both together in the same workflow.
Local models are free and private, but they have limits you’ve probably already run into.
A 9B model handles conversation and routine coding well. Ask it to debug a subtle race condition across six files, and it flounders. A 32B model does better, but it still lacks the raw reasoning depth of the largest running on thousand-GPU clusters.
A model running on your laptop has fewer , less training data, and a shorter than one backed by a datacenter. The useful question isn’t “which is better?” but “which is better for this task, right now?”
What You Get From the Cloud (And What You Give Up)
What Cloud APIs Offer
Frontier models have hundreds of billions of parameters (or more), and on complex tasks they produce better results than local models of any size.
They also handle longer context. Most frontier models now support 1M+ . You can feed an entire codebase, a book, or hours of meeting transcripts in a single prompt. Local models mostly range from 32K to around 256K, and a few open models advertise 1M. Advertised length and usable length aren’t the same thing, so test at the length you need.
On the practical side, cloud models process text, images, video, audio, and PDFs natively, while local models are still catching up. And there’s no hardware requirement at all: an API call works the same from a Chromebook as from a workstation.
What You Give Up
Your prompts and data travel to someone else’s servers. For proprietary code, medical records, legal documents, or anything covered by compliance rules, that can be a dealbreaker.
It costs money, too. Anthropic reports that Claude Code averages about $13 per developer per active day, with 90% of users under $30. Batch processing thousands of documents adds up fast. Local is free after the hardware investment.
APIs also go down, rate limits kick in, and providers change pricing or deprecate models without asking you. You can’t a frontier model (with a few exceptions), and you can’t inspect its weights. Local models work offline, don’t have rate limits, and don’t change unless you change them.
Choosing: A Decision Framework
Instead of picking one side permanently, match the model to the task.
The Quick Test
Ask three questions about whatever you’re about to do.
First, does it involve sensitive data? Proprietary code, personal information, internal documents, anything you wouldn’t paste into a public website. If yes, lean local. Some cloud providers offer data processing agreements and SOC 2 compliance, but “it never leaves my machine” is simpler to reason about.
Second, does it need deep reasoning or very long context? Architecture decisions, complex debugging, analyzing a 500-page document, creative work where quality really matters. If yes, lean cloud. These are where frontier models earn their cost.
Third, is it high-volume or repetitive? Summarizing a folder of reports, classifying a thousand emails, generating boilerplate. If yes, lean local. Repetitive work at scale is where local models save the most money.
Task-by-Task Guide
| Task | Local | Cloud | Recommendation |
|---|---|---|---|
| Simple refactoring | Great | Great | Local (free) |
| Boilerplate / CRUD | Great | Great | Local (free) |
| Test writing | Good | Great | Local for routine, cloud for tricky edge cases |
| Bug fixing (straightforward) | Good | Great | Either works |
| Multi-file architecture changes | Fair | Excellent | Cloud |
| Complex debugging | Weak | Excellent | Cloud |
| Working in an unfamiliar framework | Weak | Excellent | Cloud |
| Summarization (batch) | Good | Great | Local (free at scale) |
| Creative writing | Good | Great | Cloud if quality is critical |
| Sensitive/proprietary data | Required | Risky | Local |
For routine tasks, local is fine. For the hard stuff, cloud is usually worth the cost. Most work falls somewhere in between.
How Pricing Works (And How to Control It)
Cloud models charge per token. Every word you send and every word you get back costs money. Understanding the pricing model helps you avoid surprises.
The Three Tiers
Providers offer models at different price-performance points. The names change, the pattern doesn’t:
| Tier | What It’s For | Typical Cost (per million tokens) |
|---|---|---|
| Flagship | Hardest problems. Architecture, complex analysis. | $4-10 input, $20-50 output |
| Workhorse | Daily use. Coding, writing, general tasks. | $0.50-3 input, $3-15 output |
| Budget | High volume. Classification, formatting, simple Q&A. | $0.10-0.50 input, $0.40-2 output |
Prices change fast. Anthropic’s flagship Opus tier went from $15 input and $75 output per million tokens (Opus 4.1) to $4 and $20 (Opus 5.5). The ranges above fit Anthropic’s current list prices; the Frontier Models Reference has the current numbers for each provider. The ratios between tiers are more stable than the absolute numbers.
The Most Common Mistake
Using the flagship model for everything. Most requests don’t need the strongest model. A classification task, a format conversion, a simple Q&A: the budget tier handles these fine. Save the flagship for the 10% of tasks that actually need it.
A typical cost-effective split:
70% of requests → Budget tier ($0.10-0.50/M input)
20% of requests → Workhorse tier ($0.50-3/M input)
10% of requests → Flagship tier ($4-10/M input)
Using the midpoint of each input range, that blend costs under a fifth of what sending everything to the flagship tier would.
Combining Local and Cloud
A hybrid setup routes requests to the right model based on what the task needs.
The Simple Version
Use two tools. A local model for everyday work, a cloud model for the hard stuff.
Daily coding (refactoring, tests, boilerplate):
→ local tool (Ollama) + a 9B local model
→ Cost: $0
When you hit a wall (architecture, complex bugs, unfamiliar code):
→ switch to a frontier coding API or multimodal cloud API
→ Cost: pay-per-use, only when you need it
Cloud spend drops roughly in proportion to how much work you move to local, and the tasks that move are the ones where a 9B model does fine.
Programmatic Routing
If you’re building an application, you can route automatically. The logic doesn’t need to be complicated: check the task type (classify/format/summarize gets local, analyze/debug/architecture gets cloud), check prompt length (very long context goes to cloud), and default to local for everything else.
Latency is another criterion worth building in. Local models are deterministic in latency: your hardware is the only variable. Cloud APIs have variable latency from network round-trips and provider load. For real-time or user-facing applications, this often matters more than quality or cost.
More sophisticated routing uses a small classifier model to judge prompt complexity before routing. RouteLLM is a research library from LMSYS that routes each prompt to a strong or a weak model based on a quality threshold. Check its repo for maintenance status before depending on it. OpenRouter is a unified API gateway that lets you call many providers through one endpoint, useful when you want to switch models without changing code. It also has an Auto Router (openrouter/auto) that picks a model for each request.
The Ollama Compatibility Advantage
Because Ollama exposes an OpenAI-compatible API (Module 6 covered this), swapping between local and cloud often means changing a base URL and model name. Your application code, prompt templates, and tool definitions stay the same. If you used the OpenAI-compatible API pattern from Module 6, you can swap local and cloud with one line change: just point the endpoint URL at localhost instead of the cloud provider, or vice versa.
Making It Cheaper: Caching, Batching, Routing
If you’re using cloud APIs at any volume, these three techniques cut costs.
Prompt Caching
If you send the same system prompt on every request (and you probably do), you’re paying full price for it every time. Prompt caching lets you pay once and reuse. With Anthropic, you opt in with a cache_control field, either once at the top level of the request or on individual content blocks. Writing a 5-minute cache entry costs about 1.25x the normal input price (2x for a 1-hour entry), and reading from it costs 0.1x or less. For a system making 100 calls with the same 3,000-token system prompt, that’s roughly a 90% reduction on the system prompt portion.
Gemini applies implicit caching by default on its current models, so you get the savings without changing anything. Either way, if you’re making repeated calls with a stable prefix, keep the stable part at the start of the prompt.
Batch API
When you don’t need an immediate response, batch APIs let you submit many requests at once and get results later. Anthropic’s Batch API takes 50% off both input and output tokens, and it’s worth checking whether your provider has an equivalent.
Good candidates: nightly report generation, bulk document classification, processing a queue of support tickets, running evaluation test suites.
Context Management
Every token costs money. Common waste patterns:
- Sending an entire file when you need 10 lines
- Including 50 turns of conversation history when the last 5 are relevant
- Embedding full documentation in every request instead of using to fetch what’s needed
Trim your context before sending. Summarize old conversation turns. Use retrieval instead of stuffing everything into the prompt. A request that sends 2,000 tokens instead of 20,000 costs 10x less and often gets the same quality answer.
Next Steps
-
Module 8: Supercharge Your Setup — The MCP ecosystem, skills, plugins, memory tools, and how to discover them.
-
Module 9: Go Further — Fine-tuning, media generation, evaluation, and what’s coming next.
-
Module 6: Build Custom Tools — If you skipped ahead: APIs, function calling, MCP wiring, and RAG pipelines.
-
Model Reference — Current model recommendations for every use case and budget.
-
Hardware Reference — Upgrade paths if local performance is your bottleneck.