What's Next
What’s Changing and Why It Matters to You
This module covers what’s coming in local AI: hardware improvements, software changes, and research directions, so you know where to look as the field moves.
Everything here is backed by published papers or working code. Some of it you can use today. Some is still a research result or a community project, and it may or may not reach the tools you use soon. The direction is the same either way: the hardware you own keeps getting more capable without you buying anything.
Compression Is Getting Smarter
Why This Matters Most
Module 2 covered : compressing models from 16-bit to 4-bit so they fit in less memory. Module 3B explained the mechanics. But quantization has been hitting a floor. Below Q4, quality drops fast. The curve is flat from down to about Q5, then falls off a cliff below Q3.
Several new techniques push that cliff lower.
TurboQuant: Rotating Before Compressing
doesn’t compress model weights. It compresses the , the memory that grows as your conversation gets longer. Module 3B covered the mechanics.
Google reports that it quantizes the KV cache to 3 bits without training or fine-tuning, and cuts KV cache memory by at least 6x in its tests. At that ratio, the 8+ GB a 128K context takes on a 9B model would drop to around 1.5 GB. That would make much longer possible on hardware that can’t support them today. It’s a research result, so check the llama.cpp and Ollama release notes for official support before counting on it. More on this in our Hardware Reference.
Weight Quantization Below 4 Bits
applies a related idea to model weights themselves. Instead of quantizing each weight independently, it uses incoherence processing (a randomized transformation, a related idea to TurboQuant’s rotation) to spread information across weight dimensions before compressing. This pushes usable quantization to 2 bits.
The practical result: models that need 32 GB at Q4 would need about 16 GB at Q2. Models that need 64 GB would need 32 GB. Each bit of quantization improvement roughly halves the memory requirement. The quality tradeoff isn’t zero, but it’s shrinking with every new technique.
What This Means For Your Machine
If you have 16 GB today and run a 9B model at Q4, these techniques could let the same hardware run a 14-20B model once they’re widely supported. With 32 GB, the equivalent would be 32-40B models. Those are rough estimates, not benchmarks, but none of it requires buying anything new.
Context Windows Are Getting Longer
Many local models support 32K to 256K of context, and a few advertise 1M. already handle 1M+. Local models could support long context in theory, but they run out of memory before they run out of capability. The KV cache grows linearly with context length, and at 128K tokens on a 9B model, it eats 8 GB on its own.
KV cache compression (the TurboQuant technique above) is what unlocks this. With 3-bit quantization, rough estimates put 256K context within reach on a 32 GB machine and 512K on 64 GB.
At 32K tokens, you can fit a few files. At 128K, a medium codebase. At 512K, an entire project with its documentation. The difference between “summarize this file” and “answer questions about this entire repository” is context length, and memory is the main thing holding local models back.
On-Device Inference: Models That Don’t Fit in RAM
The Idea
Apple’s “LLM in a Flash” research asks: what if models didn’t have to fit entirely in memory?
Normally, you load all model weights into RAM before generating a single token. A 70B model at Q4 needs ~42 GB in memory. If you have 32 GB, you can’t run it.
Flash inference streams weights from SSD as needed. An model activates only a fraction of its experts per token. The inactive experts can stay on disk and stream into memory only when the router calls them.
A community project, flash-moe, applies the idea on a Mac. It streams the experts of Qwen3.5-397B-A17B, a 397B-parameter MoE model, from SSD and reports about 4.4 tokens per second on a MacBook Pro with an M3 Max and 48 GB. The model is about 209 GB on disk at 4-bit, far larger than that machine’s RAM.
What Limits It
SSD bandwidth is the new bottleneck. A fast NVMe drive reads at 5-7 GB/s. That’s 10-15x slower than DDR5 RAM (~89 GB/s) and 100x slower than an RTX 4090’s GDDR6X (~1,008 GB/s). Token generation will be slower than in-memory inference, but the question shifts from “can I run this at all?” to “how fast?”
For MoE models specifically, the hit is smaller because you’re only streaming the active experts (say 10-20% of total weights) while the shared layers stay resident in RAM.
If SSD streaming reaches the tools you use, the question shifts from “do I have enough RAM?” to “do I have enough fast storage?” NVMe drives at 5-7 GB/s are already fast enough for some model sizes.
Tool Use Is Becoming Native
What Changed
Getting a local model to use tools reliably used to require careful prompt engineering and custom parsing code. The model would sometimes forget the tool format, hallucinate tool names, or fail to parse results.
Recent open model families such as Qwen, Gemma, and Nemotron support function calling. The model knows the protocol: describe available tools in the system prompt, request a tool call with structured arguments, receive the result, and continue. No prompt hacking required.
Why MCP Matters Here
turns this from “tools work with one specific application” to “tools work everywhere.” An MCP server for GitHub written for Claude Code also works with OpenCode, Cursor, and any other MCP-compatible client. Write the integration once, use it from any .
Every new MCP server benefits every compatible agent, and Anthropic counted more than 10,000 active public MCP servers in December 2025, which shows how fast this grows when there’s a shared standard.
For local AI specifically, this means a 9B model running on your laptop can now search your files, read your database, send Slack messages, and manage GitHub issues, all through the same protocol.
Research Worth Following
| What | Why It Matters For Local AI | Where to Read |
|---|---|---|
| MoE at small scale | MoE used to be for 100B+ models. Now appearing at 9B and 3B. Gives you speed of a tiny model with knowledge from a larger set. | Mixture of Experts Explained (HuggingFace) |
| Hybrid Mamba-Transformer | Combines transformer attention (good at reasoning) with Mamba (good at long sequences, linear memory). Handles long context without quadratic cost. | Mamba (Gu & Dao, 2023), Nemotron 3 (NVIDIA) |
| Speculative decoding | Tiny draft model predicts tokens, large model verifies in batch. 2-3x speedup, same quality. Free performance. | Leviathan et al., 2023 |
| as default | Models that accept images, audio, and text as input by default, rather than as add-on capabilities. This matters for tool use and agent workflows: an agent that can read screenshots, process audio recordings, and handle PDFs alongside text can act on a much wider range of real-world inputs. | Qwen 3.x (Alibaba) |
| improvements | Small models learning from large ones, so a small model often does what a much larger one did a generation earlier. Your hardware gets more capable without upgrades. | Hinton et al. (2015), “Distilling the Knowledge in a Neural Network” |
Foundational Papers
If a concept from this curriculum made you curious, these are where the full explanations live.
| Concept | Paper |
|---|---|
| How transformers work | Attention Is All You Need (Vaswani et al., 2017) |
| How agents reason and act | ReAct (Yao et al., 2022) |
| How LoRA works | LoRA (Hu et al., 2021) |
| How works | RAG (Lewis et al., 2020) |
| How KV cache compression works | TurboQuant (Zandieh et al., 2025) |
| How linear-time models work | Mamba (Gu & Dao, 2023) |
Unfamiliar with a term? Every technical term used across this curriculum has a definition. Click any dotted-underline term on any page to see a quick definition in place.
What You Can Do Now
In Module 1 you ran your first model with a single command. That was the starting point. From there, the curriculum worked through how models actually work, how they fit on your hardware, how to get useful output from them, how agents and tools operate, how to build your own pipelines, and how to push further with fine-tuning and media generation.
You can now run and evaluate models for specific tasks rather than relying on leaderboards. You can build RAG pipelines on your own documents, so a model can answer questions from your notes, your codebase, or your organization’s internal knowledge. You can set up hybrid local-cloud workflows that use local models for routine work and route harder tasks to cloud APIs only when needed. You can extend agents with MCP tools so they can reach your databases, code repos, and communication tools. And you can fine-tune a model on custom data when prompting alone isn’t getting you consistent results.
None of this is theoretical. Local AI is genuinely useful today for real work: coding, research, document processing, automation. The people doing it daily are on r/LocalLLaMA and HuggingFace, and in the communities linked in Module 8. The field moves fast enough that staying in those spaces matters more than any single module. What you’ve built here is a foundation, not a finish line.
Where to Go From Here
That’s the full curriculum. The reference pages below are kept up to date and are worth bookmarking.
-
Model Reference — Current model recommendations for every hardware tier and use case. Check this when you’re ready to try something new or when a model recommendation from the curriculum feels dated.
-
Hardware Reference — The full hardware guide: memory equations, bandwidth tables, GPU buying advice, and the hardware calculator.
-
Sources — Every paper, tool, and resource cited across all 10 modules, organized by module.
If you want to revisit specific topics, the modules stand on their own. Module 3A and Module 3B are good references when you want to understand why something behaves the way it does. Module 5 through Module 8 cover the practical ecosystem in depth.