Running LLMs Locally in 2026: A Practical Guide for People Who Don't Want to Send Their Code to the Cloud

I started running models locally for a dumb reason: I was on a flight with no wifi and wanted to keep working on a side project that needed an LLM in the loop. I expected it to be a novelty, something I'd use for twenty minutes and abandon once I landed. Eighteen months later I've got Ollama running permanently on a home server, and a meaningful chunk of my day-to-day tooling — a commit message generator, a local RAG setup over my own notes, a first-pass code reviewer that runs before anything touches a PR — never leaves my network. Not because local models are better than Claude or GPT-4 class models, they're not, but because for a specific category of task, "good enough and instant and free and private" beats "best available but costs money and leaves my machine" every time.

The hardware question, answered honestly

The single most common question I get is some version of "what GPU do I need." The honest answer is: it depends entirely on which models you actually want to run, and most people overestimate what they need because they're picturing training, not inference.

For inference, the binding constraint is almost always VRAM (or unified memory on Apple Silicon), not compute. A model needs to fit in memory, full stop, or you're paying an enormous latency tax swapping layers to system RAM or disk. Rough rule of thumb: a model needs roughly the same number of gigabytes of VRAM as its parameter count in billions, at 8-bit quantization, and about half that at 4-bit.

On an Apple Silicon Mac — and this is genuinely the best entry point for most developers right now because of unified memory — a 16GB MacBook Air can comfortably run a 7B model at 4-bit quant with room to spare. A 32GB or 36GB machine opens up 13B-14B models comfortably and 20B-30B models at more aggressive quantization. My daily driver is an M2 with 32GB, and I run Llama 3.1 8B for quick tasks and Qwen2.5-Coder 14B when I want better code quality, with both fitting comfortably.

If you're on a discrete GPU setup, an RTX 4090 with 24GB VRAM will run most 13B models at full precision and 30B+ models quantized, which covers the vast majority of genuinely useful open models as of today. You do not need an A100 or an H100 for local inference — those matter for training and for serving many concurrent users, neither of which applies to a single developer running things locally.

Getting started with Ollama

Ollama is the easiest on-ramp by a wide margin, and it's what I'd point anyone new at first, because it handles model download, quantization selection, and a local API server with almost no configuration:

curl -fsSL https://ollama.com/install.sh | sh

ollama pull llama3.1:8b
ollama run llama3.1:8b

That second command drops you into an interactive chat, but the more useful part for actual development work is that Ollama exposes an OpenAI-compatible API on localhost:11434 by default, so you can point existing tooling at it with minimal changes:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Write a regex that validates a US phone number",
  "stream": false
}'

Or, using the OpenAI-compatible endpoint, which is what makes Ollama genuinely useful in an existing codebase rather than just a chat toy:

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama',
});

const response = await client.chat.completions.create({
  model: 'qwen2.5-coder:14b',
  messages: [{ role: 'user', content: 'Refactor this function to be pure' }],
});

That drop-in compatibility is a big deal in practice — I've swapped the base URL on internal tools between a local model and a cloud API with a single environment variable change more times than I can count, usually to compare quality or to keep working when I'm somewhere with bad internet.

Quantization, actually explained

Quantization reduces the precision of a model's weights to shrink its memory footprint, and understanding the trade-off is the difference between picking a model that's usably fast and one that's either too slow or too dumb for what you need.

Full precision (FP16) is the reference quality but the heaviest. Quantization schemes you'll see constantly in Ollama and llama.cpp model names — Q8_0, Q5_K_M, Q4_K_M, Q3_K_S — trade bits per weight for size and speed. In my experience, Q4_K_M is the sweet spot for almost everything: it's roughly a quarter the size of FP16, runs noticeably faster, and the quality degradation versus full precision is small enough that I genuinely can't reliably tell the difference on coding or writing tasks in a blind comparison. Below Q4, into Q3 and Q2, degradation becomes obvious — I've seen models start hallucinating function signatures and losing coherence on longer contexts at Q2_K, which makes sense since you're down to roughly 2 bits per weight and a lot of nuance just isn't representable anymore.

ollama pull llama3.1:8b-instruct-q4_K_M

For more control than Ollama gives you, llama.cpp is the underlying engine (Ollama actually wraps it) and lets you tune context length, GPU layer offloading, and batch size directly:

./llama-server \
  -m models/qwen2.5-coder-14b-instruct-q4_k_m.gguf \
  --n-gpu-layers 999 \
  --ctx-size 8192 \
  --port 8080

--n-gpu-layers 999 just means "offload as many layers to GPU as will fit" — llama.cpp will cap it automatically at what your VRAM supports, which is genuinely convenient when you're not sure of the exact ceiling.

Model selection for real developer tasks

For code-specific work, Qwen2.5-Coder and DeepSeek-Coder-V2 have both genuinely impressed me — Qwen2.5-Coder 14B at Q4 quant handles routine refactors, test generation, and code review comments about as well as GPT-3.5-class models did, running entirely on a laptop with no API cost. For general reasoning and writing, Llama 3.1/3.3 and Mistral's newer releases are solid defaults. If you need long context for RAG over a large codebase or document set, check the specific model's trained context length before assuming — a lot of open models claim large context windows but degrade meaningfully past 8K-16K tokens in practice, which is a gap between the spec sheet and reality that's easy to miss until your retrieval quality mysteriously drops on longer documents.

Where local genuinely wins, and where it doesn't

Local wins decisively on three axes: privacy (nothing about proprietary code, customer data, or unreleased product plans leaves your machine — this alone is why I moved my code review pass local, since I didn't want a first-pass reviewer sending diffs of unreleased features to a third party), cost at volume (once you're making thousands of API calls a day for something like commit message generation or log summarization, local inference is free after the hardware, versus a real ongoing API bill), and latency for small models on tasks that don't need frontier reasoning (a 7B model answering a quick lookup question locally beats a network round trip to any cloud API on raw response time).

Local loses, still, and I don't think this changes soon, on genuinely hard reasoning, on tasks requiring the largest context windows, and on anything where being right matters more than being fast or free. I don't use local models for architecture decisions, for debugging a gnarly distributed systems issue, or for anything customer-facing where a wrong answer has real cost — that's still Claude or GPT-4 class territory, full stop, and I don't see open local models closing that specific gap in the near term even as they keep improving on everything else.

The practical setup I've landed on, and would recommend to anyone starting out, is running Ollama as a background service and defaulting to it for routine, high-volume, low-stakes tasks — commit messages, quick code lookups, drafting, summarization — while keeping Claude Code or a cloud API in the loop for anything that actually requires careful reasoning or where a wrong answer costs real time to catch. Treating it as an either-or choice is the wrong frame entirely. It's a routing decision, and once you start thinking about it that way, the question stops being "should I run models locally" and becomes "which of my daily tasks don't actually need the best model in the world," which turns out to be a lot more of them than I expected before I actually tried it.