ReferenceReference

🖥️Hardware Reference

Hardware Reference

Last verified: October 2026. Hardware requirements shift as quantization improves and model architectures change (MoE models need more RAM but less compute). Model sizes below are Ollama download sizes checked in October 2026. If something looks outdated, check the Model Reference.


Quick Answer: What Do I Need?

Your GoalMinimumRecommendedSweet Spot
Try it out (basic chat)8 GB RAM16 GB RAMAny modern machine
Daily driver (good quality)16 GB RAM32 GB RAMApple M-series or NVIDIA GPU
Power user (large models)32 GB RAM64 GB RAMM5 Pro/Max, or a used RTX 3090 / RTX 5090
Enthusiast (frontier-local)64 GB RAM128 GB+M5 Max, M5 Ultra or M3 Ultra Mac Studio, or multi-GPU

Already have a machine? Use our Hardware Calculator to see exactly which models fit your specs.


Hardware Tiers

Tier 1: Entry (8 GB RAM)

What you can run: Small models, 1B to 4B parameters at Q4 quantization.

Experience: Solid for simple chat, Q&A, basic summarization. Slower on complex reasoning. Like a quick assistant with broad but shallow knowledge.

Typical hardware:

  • Older MacBook Air (M1/M2, 8 GB)
  • Budget Windows laptops
  • Older desktops with DDR4

Best models at this tier:

ModelDownload SizeOllama Command
Qwen 3.5 4B3.3 GBollama pull qwen3.5:4b
Nemotron 3 Nano 4B2.8 GBollama pull nemotron-3-nano:4b
Phi-4 Mini (3.8B)2.5 GBollama pull phi4-mini

RAM needed is roughly the download size plus 4 GB for the OS, runtime, and KV cache, rounded up. The Model Reference explains the rule.


Tier 2: Standard (16 GB RAM)

What you can run: Medium models, 7B to 9B at Q4, or 4B models at Q8 (near-lossless).

Experience: Good conversations, useful coding assistance, moderate reasoning. A good baseline for most users.

Typical hardware:

  • MacBook Air/Pro, M1 through M5 (16 GB)
  • Mid-range Windows laptops
  • Desktops with DDR4/DDR5

Best models at this tier:

ModelDownload SizeOllama Command
Qwen 3.5 9B6.6 GBollama pull qwen3.5:9b
Gemma 4 12B7.7 GBollama pull gemma4:12b
Gemma 4 E4B6.6 GBollama pull gemma4:e4b
Phi-4 (14B)9.1 GBollama pull phi4

Tier 3: Power (32 GB RAM)

What you can run: Large models, 14B to 32B at Q4, or 9B at Q8. MoE models like Qwen 3.6 35B-A3B fit.

Experience: Near-frontier quality for many tasks. Complex reasoning, detailed code generation, nuanced writing. Local models at this tier can handle most of what you’d otherwise reach for a cloud API to do.

Typical hardware:

  • MacBook Pro with M5 Pro (24 to 64 GB configurations)
  • Gaming desktops with 32 GB DDR5
  • NVIDIA RTX 3060/4060 (12 GB VRAM) + 32 GB system RAM

Best models at this tier:

ModelDownload SizeOllama Command
Qwen 3.8 27B18 GBollama pull qwen3.8:27b
Qwen 3.6 35B-A3B (MoE)23 GBollama pull qwen3.6:35b
Gemma 4 31B19 GBollama pull gemma4:31b
GPT-OSS 20B (MoE)14 GBollama pull gpt-oss:20b

Tier 4: Enthusiast (64 GB RAM)

What you can run: Very large dense models, 70B at Q4, or 27B to 32B at Q8. The 120B-class MoE models (Qwen 3.5 122B-A10B, Nemotron 3 Super) do not fit here. They need the 128 GB tier below.

Experience: Excellent quality for most tasks. Going beyond 64 GB mainly pays off for specialized workloads.

Typical hardware:

  • MacBook Pro with M5 Pro (64 GB) or M5 Max, or a Mac Studio
  • High-end desktops with 64 GB DDR5
  • A 24 GB or 32 GB NVIDIA card (used RTX 3090, RTX 5090) + 64 GB system RAM

Best models at this tier:

ModelDownload SizeOllama Command
Qwen 3.8 27B at Q830 GBollama pull qwen3.8:27b-q8_0
Llama 3.3 70B (Dec 2024)43 GBollama pull llama3.3:70b
DeepSeek R1 70B (older distill)43 GBollama pull deepseek-r1:70b

Tier 5: Extreme (96 GB+ RAM)

What you can run: The 100B+ MoE models: GPT-OSS 120B (65 GB), Llama 4 Scout (67 GB), Nemotron 3.5 Super (80 GB), Qwen 3.5 122B-A10B (81 GB), Nemotron 3 Super (87 GB). At 256 GB and up, Qwen 3 235B-A22B and Llama 4 Maverick. The newest open flagships (DeepSeek V4, Kimi K3, GLM-5.3) are far larger and are mostly offered as cloud-hosted models.

Experience: Closer to cloud frontier models than anything at the lower tiers. For most people this is more capability than they need.

Typical hardware:

  • Mac Studio with M5 Max (128 GB), M5 Ultra (up to 512 GB) or M3 Ultra (up to 512 GB)
  • NVIDIA DGX Spark (64 or 128 GB unified memory)
  • Multi-GPU workstations (2× RTX 3090 = 48 GB VRAM)
  • High-end server hardware

Best models at this tier:

ModelDownload SizeOllama Command
GPT-OSS 120B (MoE)65 GBollama pull gpt-oss:120b
Llama 4 Scout (109B MoE)67 GBollama pull llama4:16x17b
Nemotron 3.5 Super (120B MoE)80 GBollama pull nemotron-3.5-super
Qwen 3.5 122B-A10B (MoE)81 GBollama pull qwen3.5:122b-a10b
Nemotron 3 Super (120B MoE)87 GBollama pull nemotron-3-super
Qwen 3 235B-A22B (MoE)142 GBollama pull qwen3:235b-a22b
Llama 4 Maverick (400B MoE)245 GBollama pull llama4:128x17b

Understanding Memory: RAM, VRAM, and Unified

System RAM (all machines)

Your computer’s main memory. This is where models load when you don’t have a dedicated GPU, or when the model is too large for GPU memory.

  • DDR4: Standard in pre-2023 machines. Slower bandwidth (~50 GB/s for dual-channel).
  • DDR5: Current standard. Faster (~89 GB/s dual-channel), better for LLMs.
  • How to check: Mac → About This Mac. Windows → Settings → System → About. Linux → free -h.

GPU VRAM (NVIDIA/AMD)

Dedicated memory on your graphics card. Much faster than system RAM for model inference.

  • NVIDIA GeForce: RTX 3060 (12 GB), RTX 5060 Ti (16 GB), RTX 3090 (24 GB, used), RTX 5090 (32 GB). The RTX 4090 (24 GB) is no longer made, so it only turns up used.
  • NVIDIA Pro: A6000 (48 GB), H100 (80 GB)
  • How to check: nvidia-smi in terminal, or NVIDIA Settings → System Information

Partial offloading: If a model is too large for your VRAM, part loads on the GPU (fast layers) and the rest stays in system RAM (slower layers). This is automatic in Ollama and llama.cpp. You’ll see something like “32 of 48 layers offloaded to GPU” in the logs.

Apple Silicon Unified Memory

Apple M-series chips share a single memory pool between CPU and GPU. There’s no separate VRAM. It’s all unified.

Why this matters for LLMs: On a PC, if your model exceeds VRAM, performance falls off a cliff as layers spill to DDR. On Apple Silicon, all memory is equally fast, so there’s no bottleneck between GPU and CPU memory. A Mac with 64 GB of unified memory can run a model that would spill out of a 24 GB GPU into slow system RAM on a PC.

Apple ChipMax Unified MemoryMemory Bandwidth
M432 GB120 GB/s
M4 Pro64 GB273 GB/s
M4 Max128 GB410 or 546 GB/s
M3 Ultra512 GB819 GB/s
M532 GB153.6 GB/s
M5 Pro64 GBup to 307 GB/s
M5 Max128 GB460 GB/s (32-core GPU) or 614 GB/s (40-core GPU)
M5 Ultra512 GBabout 1.2 TB/s

There is no M4 Ultra. Apple skipped it: the top chip between the M4 and M5 generations is the M3 Ultra (Mac Studio, 2025). The M5 Ultra arrived in the Mac Studio in August 2026. The Mac Studio starts at $2,499 with the M5 Max and $5,499 with the M5 Ultra. The 512 GB configuration was scheduled for late October 2026. Bandwidth varies with GPU core count within a chip, so check the exact configuration before you buy.


Memory Bandwidth: The Hidden Speed Limit

Two machines with identical RAM can run the same model at vastly different speeds. The bottleneck is : how fast the processor can read model weights from memory.

Token generation is bandwidth-bound: the processor spends most of its time waiting for numbers to arrive from memory, not doing math. More bandwidth = more tokens per second.

Bandwidth by Hardware Type

HardwareBandwidth7B Q4 Speed70B Q4 Speed
DDR4 laptop (2-channel, 3200 MT/s)~50 GB/s~8 tok/sToo slow
DDR5 desktop (2-channel, 5600 MT/s)~89 GB/s~15 tok/s~2 tok/s
DDR5 workstation (8-channel)~358 GB/s~50 tok/s~8 tok/s
Apple M5 Pro (unified)up to 307 GB/s~50 tok/s~5 tok/s
Apple M5 Max (unified)up to 614 GB/s~100 tok/s~10 tok/s
Apple M3 Ultra (unified)819 GB/s~135 tok/s~13 tok/s
NVIDIA RTX 5090 (GDDR7)~1,792 GB/s~290 tok/sGPU only if fits

Token speeds are estimates, not measurements: bandwidth divided by model size (about 4 GB for 7B Q4, about 40 GB for 70B Q4), times roughly 0.65 for real-world overhead. Actual rates vary with model architecture, context length, and system load.

Key Insights

Channels matter more than DDR generation. A 12-channel DDR4 server (~307 GB/s) is faster than a 4-channel DDR5 desktop (~192 GB/s). Bumping DDR5 speed from 4800 to 6000 MT/s gives about a 20% speedup.

The M5 Max delivers up to 614 GB/s in a laptop (460 GB/s with the 32-core GPU). Matching that on a PC requires a high-end GPU or a multi-channel workstation.

GPU VRAM is the fastest. If a model fits entirely in your GPU’s VRAM, you get the full GPU bandwidth. The moment it spills to system RAM, your speed is limited by the slower DDR bandwidth for those layers.


The Memory Equation

When you’re checking if a model fits, here’s the full picture:

Total RAM needed  =  Model Weights  +  KV Cache  +  OS/Apps  +  Overhead
ComponentSizeNotes
Model weightsFile size you downloadedThe largest component
KV cacheGrows with conversation256K context on a 9B model ≈ 2-4 GB
OS + other apps3-4 GBMore if you’re running a browser, code editor, etc.
Overhead~10% of model sizeBuffers, scratch space, llama.cpp internals

Quick Rule

Model file size should be ≤ 75% of your available RAM (total RAM minus OS/apps). This leaves room for KV cache and overhead.

Total RAMAvailableMax Comfortable ModelExample
8 GB~5 GB~3.5 GBQwen 3.5 4B at Q4
16 GB~12 GB~9 GBQwen 3.5 9B at Q4
32 GB~28 GB~20 GBQwen 3.8 27B at Q4 (18 GB)
64 GB~60 GB~45 GBLlama 3.3 70B at Q4 (43 GB)
128 GB~120 GB~90 GBQwen 3.5 122B-A10B at Q4 (81 GB)

MoE Models: The Catch

models load ALL their parameters into memory, even though only a fraction activate per token. A 235B MoE with 22B active still needs ~142 GB. You get 22B compute speed but 235B memory cost. That 22B of active compute draws on 235B of knowledge, which is why output quality is higher than the active count suggests.


Inference Engines by Platform

Different engines work best on different hardware. Here’s what to use where:

Your HardwareBest EngineWhy
Apple Silicon (any)Ollama or MLXUnified memory, Metal GPU acceleration
NVIDIA GPU (fits in VRAM)Ollama or vLLMFull GPU acceleration via CUDA
NVIDIA GPU (partial offload)Ollama or llama.cppSplit layers between GPU and CPU
CPU only (Intel/AMD)OllamaAVX2/AVX-512 optimizations in llama.cpp
Multi-GPUvLLM or llama.cppTensor parallelism across GPUs

All engines use files (except vLLM which also supports SafeTensors). A model downloaded for one engine works with any other.

New to this? Just use Ollama. It auto-detects your hardware and configures GPU offloading. See Module 1 for setup.


GPU Buying Guide

If you’re considering a GPU purchase specifically for local AI:

GPUVRAMBandwidthStatusBest For
RTX 5060 Ti 16GB16 GB~448 GB/sCurrentBudget. Fits 14B at Q4.
RTX 309024 GB936 GB/sUsed onlyBest value per GB of VRAM. Fits 27B at Q4.
RTX 409024 GB1,008 GB/sNo longer made, used onlyFast, but priced above a 3090 for the same VRAM.
RTX 509032 GB~1,792 GB/sCurrentFastest consumer card. Fits 27B to 32B at Q4 with room for context.
RTX A600048 GB768 GB/sOlder pro cardFits 70B at Q4.

GPU and memory prices have swung a lot through 2026, and street prices for the RTX 5090 have run well above list. Check live listings before you plan a budget around any number you read here.

AMD

AMD GPUs work with llama.cpp via ROCm, but NVIDIA’s CUDA ecosystem is more mature and better optimized for LLMs. If you already have an AMD GPU, it will work. If you’re buying specifically for AI though, NVIDIA is the safer bet.

Intel

Intel Arc GPUs have experimental SYCL support in llama.cpp. Not recommended as a primary LLM GPU yet.

Other unified-memory boxes

NVIDIA’s DGX Spark is a small desktop with 64 GB or 128 GB of unified memory at 273 GB/s. That fits large MoE models, but the bandwidth is closer to an M5 Pro than to a GPU, so token generation is slower than on a discrete card. NVIDIA sells it through its marketplace and partners.

When NOT to Buy a GPU

  • You have Apple Silicon with 32 GB+ unified memory. The bandwidth is already excellent and everything just works. A GPU won’t help. Macs don’t use external GPUs for LLM inference.
  • You just want to chat casually. A 9B model on CPU with 16 GB RAM is fast enough for conversation.
  • You’d be better off with more RAM. On a DDR5 desktop, going from 32 GB to 64 GB of RAM (to fit a 70B model) can be cheaper and more versatile than adding a GPU, though check current RAM prices first.

Emerging Technologies

Things that are changing the hardware equation right now:

TurboQuant (Google)

Compresses the to 3 bits with no accuracy loss. That’s a 6x reduction in context memory.

Why it works: Normal quantization (rounding numbers to fewer bits of precision) performs badly on the vectors inside transformer attention, because those vectors tend to have one or two massive values and everything else near zero. When you round a vector like [0.0001, 0.9999, 0.0003], the big value snaps to 1 and the rest snap to 0. You lose most of the information.

TurboQuant’s trick: before quantizing, randomly rotate the vector in its high-dimensional space. A random rotation spreads the weight evenly across all components, so no single component dominates. Quantization works much better on evenly-distributed values. During inference, the rotation is reversed (counter-rotated) to recover the original meaning.

The paper adds a second step that corrects biases when computing attention dot products with quantized vectors, but the rotation is what makes 3-bit KV cache practical.

What it means for you: The model file stays the same size, but long conversations eat far less RAM. Community implementations exist (see the llama.cpp discussion), but support in the tools you already use varies, so check release notes before counting on it. Where it works, 128K+ context windows fit on hardware that previously ran out of memory.

LLM in a Flash (Apple)

Streams model weights from SSD instead of loading everything into RAM. Apple’s 2023 paper showed models larger than available DRAM running with acceptable speed by keeping weights in flash and loading only the parts needed. Performance depends heavily on SSD speed, and mainstream tools do not do this yet. It would change the “will it fit?” question from “do I have enough RAM?” to “do I have enough storage?”

Speculative Decoding

Uses a tiny “draft” model to predict multiple tokens, then verifies them with the large model in one pass. Can deliver 2–3x speedup with no quality loss. Available in llama.cpp (llama-server takes a draft model with -md; see its speculative decoding docs) and vLLM.


Tools


How to Check Your Specs

RAM:

  • Mac: Apple menu → About This Mac → Memory
  • Windows: Settings → System → About → Installed RAM
  • Linux: free -h (look at “total” column)

GPU and VRAM:

  • NVIDIA: Run nvidia-smi in terminal
  • Mac: Apple menu → About This Mac → displays the chip (M1 through M5, unified memory, no separate VRAM)
  • Windows: Task Manager → Performance → GPU → Dedicated GPU Memory

Memory Bandwidth:

  • Mac: Determined by your chip (see Apple Silicon table above)
  • PC: Depends on RAM type (DDR4/DDR5), speed (MT/s), and number of channels. Check with CPU-Z (Windows) or dmidecode -t memory (Linux)