How Much VRAM and RAM Do You Need for Local LLM Inference? | Memory Math

Guide

Local LLM inference needs roughly 0.6 GB of memory per billion 4-bit parameters, plus KV cache, context, and operating system overhead on top.

A 7B model file that fits in 6 GB of VRAM can still crash the moment the context passes a few thousand tokens. The weight file is only the first bill. Working out how much VRAM and RAM you need for local LLM inference means adding the KV cache, the runtime, and the operating system to the download size, then leaving several gigabytes unspent.

The table below pairs memory sizes with the model classes people run on them. Sections around it show the arithmetic behind each row, so you can recalculate for your own context length instead of trusting a sticker.

What Does a 4-Bit Model Cost in Memory?

A 4-bit model costs roughly 0.6 GB per billion parameters, which puts a 7B file near 4–5 GB and a 70B file near 40–45 GB before anything else joins the total. Weights are the biggest single line item, and the one most people stop counting at.

Quantization stores each weight at lower precision. Q4 formats use about four bits per weight; Q8 files take more memory but generally hold more fidelity. Q4_K_M is the common compromise between size and quality, but the result depends on the model and the quantizer that built it.

Reported Q4 footprints cluster tightly:

  • 7B at Q4: about 4–5 GB
  • 13B at Q4: about 8–9 GB
  • 34B at Q4: about 20–22 GB
  • 70B at Q4: about 40–45 GB

Files on disk run bigger than the arithmetic alone suggests. The gap between download size and running footprint is KV cache plus runtime overhead.

Usable Memory 4-Bit Model Class What Fits Alongside It
8 GB total VRAM or unified 3B–7B Short contexts and one app at a time
12 GB VRAM 7B–13B Moderate context, desktop still responsive
16 GB VRAM 7B–14B Longer context with runtime overhead covered
24 GB VRAM 27B–35B Or 7B–14B with large context windows
32 GB VRAM 30B-class, some 70B at aggressive quantization Depends on context length and backend
48–64 GB unified or system RAM 30B–35B, some 70B via CPU offload Slower than pure VRAM, but it loads
96–128 GB and up 70B-class at higher precision Long contexts and multi-machine setups

Each row assumes 4-bit weights and a modest context. Push either one and move up a row.

VRAM and RAM for Local LLM Inference: Where The Numbers Come From

On a discrete NVIDIA or AMD card, VRAM is the fast budget, while system RAM carries the operating system, model loading, and any offloaded layers. On an Apple silicon Mac the CPU and GPU share one unified pool, so the split vanishes and the total becomes the only number that matters.

The KV cache grows with every token kept in context. Long prompts, chat history, pasted documents, and high output limits all inflate it, which is why a model that loads fine can die mid-answer.

llama.cpp shrinks that cache through --cache-type-k and --cache-type-v, including q8_0 and q4_0. Lower KV precision buys back memory and can cost output quality, so change one side at a time and compare answers on the same prompt.

Baseline guidance puts a workable floor at 16 GB of system RAM, with at least 4 GB of dedicated VRAM on Windows machines; Apple silicon support in cited guidance wants macOS 14 or newer. A 24 GB MacBook Pro runs an 8B model in BF16 or a 30B mixture-of-experts model at 4-bit, each workload staying under 18 GB in that test, as detailed in Apple’s MLX research on unified memory.

Unified Memory, Offload, and What Breaks First

Offload keeps a model running after VRAM fills by pushing layers into system RAM, and it runs substantially slower than holding everything on the card. Unified memory skips the copying penalty, but macOS and every open app draw from the same pool.

The ceiling rises with the pool. Context and model size both get paid for in memory.

Hardware choices follow the same math. A machine with 32 GB of unified memory runs mid-size models well and large ones slowly, and our roundup of the best laptop for running LLMs covers which configurations hold that line under sustained load.

  • Treating parameter count as the whole bill; the KV cache and runtime are extra.
  • Assuming every 7B file is identical across BF16, Q8, Q5, and Q4 builds.
  • Filling a GPU to its nominal capacity and leaving nothing for drivers or the desktop.
  • Raising context length without recalculating what the cache now costs.

Set the budget in order: pick the model class from the table, subtract several gigabytes for the desktop and runtime, size the context you truly use, then cut KV precision or step down a model size if the numbers come up short. A model that fits with memory to spare beats a bigger one that pages to disk.

Common Questions

Can A 7B Model Run On 8 GB Of RAM?

Yes, at 4-bit quantization with short context. A 7B file occupies about 4–5 GB, leaving little headroom for the KV cache, the desktop, and the runtime, so keep conversations brief and expect slower generation while the operating system manages the squeeze.

Does Extra System RAM Help When The GPU Is Already Full?

It helps, though not the way a bigger GPU does. Additional RAM lets the runtime offload layers and keep a larger model loaded, and each offloaded layer costs speed because weights cross the PCIe bus. The model will work, just slowly.

Why Does A Model Crash After It Loads Without Errors?

The KV cache is usually the culprit. Loading allocates only the weights; the cache grows as tokens accumulate, and a long prompt or a high output limit can exhaust the remaining memory mid-generation. Lower the context length or quantize the cache and the crash generally stops.

Sources