How to Build a Home Lab for AI on a Budget

Running AI models locally is no longer a niche hobby. Models got smaller and better, quantization got better, and Ollama made the software side simple. The remaining question is hardware, and hardware is where people waste money.

This guide is about the minimum useful AI inference setup: the cheapest path to running local LLMs you will actually use, without buying hardware you don’t need. If you already run a home lab, you may have half the parts.

GPU Memory: The Number That Matters Most

For local LLM inference, the accelerator decides almost everything. The CPU, RAM and case exist to feed it. Two numbers matter: memory capacity (how large a model you can load) and memory bandwidth (how fast it can generate).

If you haven’t read How Much VRAM Do You Need for Local LLMs?, start there. The short version, by model size class (download sizes are Ollama’s default Q4_K_M files):

• 8B models (Llama 3.1 8B at 4.9GB, Qwen3 8B at 5.2GB): 8GB at Q4_K_M, or 12GB at Q8_0
• 14B models (Qwen3 14B at 9.3GB): 12GB at Q4_K_M, or 16GB at up to Q6_K
• 32B models (Qwen3 32B at 20GB): 24GB at Q4_K_M with about 8K of context, or 32GB at Q5 or Q6
• 70B models (Llama 3.1 70B at 43GB): about 48GB for Q4_K_M with 4K to 8K of context, more than any single consumer GPU holds

The download is not the whole requirement. The context window (the KV cache) needs memory too, so leave headroom above the file size.

Bandwidth sets the speed. Generating each token reads most of the model’s weights from memory, so speed on a dense model scales with bandwidth divided by model size. That is why a 936 GB/s RTX 3090 generates faster than a 448 GB/s RTX 5060 Ti on a model both can hold, and why an integrated GPU sharing dual-channel system memory is slower still.

That makes the budget decision straightforward. Here are the tiers.

Entry Tier: Used RTX 3060 12GB

The NVIDIA RTX 3060 12GB (Used) is a previous-generation card that remains a common low-cost route to 12GB on NVIDIA’s CUDA stack. NVIDIA lists it with 12GB of GDDR6 on a 192-bit bus, a 170W power rating and a 550W system power recommendation.

It holds 8B models at up to Q8_0 and 14B models at Q4_K_M. Anything in the 32B class has to spill layers into system RAM, which slows generation sharply. Buy it used, compare completed sales rather than asking prices, and test it before the return window closes.

24GB Tier: Used RTX 3090

The RTX 3090 (Used) is the usual route to 24GB of VRAM without buying a current flagship. NVIDIA lists 24GB of GDDR6X on a 384-bit bus, 350W of board power and a 750W system power recommendation, and the Founders Edition is a 3-slot, 313mm card.

24GB holds 32B models such as Qwen3 32B (20GB) at Q4_K_M with about 8K of context. It does not make 70B practical: a standard Q2_K file of Llama 3.1 70B is about 26.4GB in bartowski’s GGUF listing, so it does not fit. The smaller IQ2_XS file (about 21.1GB) squeezes in with little room for context and a real cost in quality.

Used prices vary widely with demand and condition. Compare completed sales rather than asking prices, and buy with a return window so you can stress-test the card.

The RTX 5060 Ti 16GB is the new-card alternative: 16GB of GDDR7 at 448 GB/s, a 180W rating and NVIDIA’s 600W system power recommendation. It runs 14B models at up to Q6_K, but 32B models do not fit. Street prices vary, so check the current price before you compare it with a used 3090.

For a head-to-head of these two cards, see RTX 5060 Ti 16GB vs RTX 3090 for local LLMs.

The Oddball: Used Tesla P40

The Tesla P40 (Used) is a 2016 Pascal datacenter card with 24GB of memory. NVIDIA’s datasheet lists 346 GB/s of bandwidth, PCIe 3.0 x16, a 250W maximum and a server-optimized design. In practice that means no display outputs and no fan of its own: it expects server-style forced airflow, so you add a fan shroud. It takes power through a CPU-style (EPS) 8-pin connector rather than a PCIe 8-pin, so plan for the right adapter or a spare EPS lead. NVIDIA’s R580 driver branch is the last to support it, and modern inference stacks target newer GPUs, so expect some configuration work.

If you have a chassis with strong airflow and you enjoy solving cooling and driver problems, the P40 gets you to 24GB on a used-card budget. It is an enthusiast path, not a beginner one.

What About AMD?

AMD GPUs work with local LLMs. Ollama supports recent Radeon cards through ROCm and other AMD and Intel GPUs through Vulkan, and llama.cpp has HIP and Vulkan backends. Expect more setup work than with CUDA, and check that your exact card is on the supported list before you buy. Read NVIDIA vs AMD for LLMs for the full picture. For a budget build where you want things to just work, NVIDIA is the safer bet.

The Build: Desktop Tower or Mini PC

Desktop Tower: The Simplest Path to a Discrete GPU

A desktop or tower workstation is the simplest way to run a full-size GPU. You need a PCIe x16 slot, enough power supply wattage and room in the case for the card’s cooler.

The rest of the system does not need to be expensive. When the whole model sits in VRAM, the GPU performs the inference computation:

Memory and SSD prices move quickly, so price these parts on the day and let the GPU take the largest share of the budget. Models load from disk into VRAM once, so a SATA SSD is fine.

Mini PCs: Integrated GPUs and Unified Memory

Mini PCs cannot take a full-size GPU, but that does not limit them to CPU inference.

Radeon 780M mini PCs. The Beelink SER9 Pro pairs a Ryzen 7 H 255 (8 cores, 16 threads) with Radeon 780M graphics and 24GB or 32GB of soldered LPDDR5X-6400. Ollama can run models on that integrated GPU through Vulkan, using system memory as VRAM. The Minisforum UM890 Pro uses the same 780M graphics with socketed DDR5 up to 96GB. These machines hold 8B and 14B models comfortably, but dual-channel system memory has a fraction of a discrete card’s bandwidth, so generation is slower.

Unified-memory machines. The Minisforum MS-S1 Max uses AMD’s Ryzen AI Max+ 395 with Radeon 8060S graphics and up to 128GB of quad-channel LPDDR5X-8000. That is enough memory to hold Llama 3.1 70B at 4-bit (43GB) with room to spare, which no single consumer GPU can do. It is not a budget machine: Minisforum lists the 128GB/2TB version at $2,919.90. Apple’s Mac mini is the other route. Apple sells the M6 with up to 32GB and the M5 Pro with 24GB, 48GB or 64GB at 307 GB/s (the Amazon listing is the 24GB model; the larger sizes are build-to-order from Apple), and Ollama runs on Apple Silicon through Metal.

A slot or an external GPU. The Minisforum MS-A2 has a PCIe x16 slot wired PCIe 4.0 x8, but it takes low-profile, slot-powered cards only, so no full-size GPUs. Mini PCs with USB4 (the SER9 Pro, the UM890 Pro) or OCuLink (the UM890 Pro’s PCIe 4.0 x4 port) can drive an external GPU dock with its own power supply. That adds cost and cabling, so compare it against a used tower.

If you already own a mini PC, install Ollama and try a small model before you buy anything. It costs nothing to test. The best mini PC for local AI guide compares the options in more depth.

Power Budget

GPU inference draws far more than the rest of a home lab. A mini PC running Proxmox idles in the low tens of watts or less; a GPU tower under load draws hundreds.

GPU Power Ratings

The last column is an estimate built from ratings: the card’s board power plus about 80 to 100W for the rest of a basic tower. Token generation often draws less than the full rating, so measure your own box with a plug-in meter.

PSU Sizing

Follow NVIDIA’s system power guidance above as the floor, and buy an 80 Plus Gold unit if the box will run for many hours a day. Leave headroom for transient spikes on the larger cards.

UPS Protection

A GPU running inference is doing real work, possibly serving an Open WebUI instance to your household. An outage mid-write can corrupt the OS drive or a model download.

The CyberPower CP1500PFCLCD is a 1500VA/1000W sine-wave UPS. CyberPower rates it at 10 minutes at half load (500W), and an RTX 3090 box under load sits near that point, so plan for a clean shutdown rather than riding out an outage. Size the whole box, not just the GPU, using how to size a UPS and best UPS for home lab.

Electricity Cost

At $0.18 per kWh (close to the US residential average in EIA’s latest data), each watt running around the clock costs about $1.58 a year:

• An RTX 3060 box averaging 120W costs about $16 a month.
• An RTX 3090 box averaging 200W costs about $26 a month.
• Either box idling at 60W costs about $8 a month doing nothing.

Whether local inference pays for itself depends on which API tier it replaces. Our home lab vs cloud guide works through the break-even.

Software Stack

The software side is simple now. Two tools cover most home lab AI use cases.

Ollama: The Engine

Ollama is the Docker of local LLMs. Install it, run ollama pull llama3.1:8b, and you have a working model in minutes. It handles downloads, quantization selection and GPU detection, and exposes a local API compatible with OpenAI’s format.

Good models to start with, by size class (sizes are Ollama’s default Q4_K_M downloads):

• 8B: Llama 3.1 8B (4.9GB), Meta’s general-purpose model, and Qwen3 8B (5.2GB) from Alibaba
• 14B: Qwen3 14B (9.3GB), a clear step up for everyday assistant work
• 32B: Qwen3 32B (20GB), for 24GB cards
• Reasoning: DeepSeek-R1 distills at 8B, 14B and 32B

The 8B models fit on any 8GB card. The 14B models want 12GB or more, and the 32B models want 24GB.

Install Ollama on Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run llama3.1:8b

Open WebUI: The Interface

Open WebUI gives you a ChatGPT-style web interface for your local Ollama instance. It supports conversation history, multiple models, document upload (RAG) and user accounts if you want to share the setup with family or coworkers.

Run it with Docker:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

This connects to Ollama on the host. Open http://localhost:3000 and you have a private, self-hosted assistant.

Other Tools Worth Knowing

• text-generation-webui (oobabooga) is more complex than Ollama but gives fine-grained control over sampling settings and extensions.
• vLLM is a high-throughput serving engine. It is overkill for one person but useful when several users or an application share the model.
• LocalAI is an OpenAI-compatible server with several backends, handy as a drop-in replacement for OpenAI’s API in existing tools.

For most home labs, Ollama plus Open WebUI is the whole stack.

What Can You Actually Run?

Here is what each memory tier holds, using llama.cpp’s quantization levels (Q4_K_M is the usual balance of quality and size):

In our judgment the step from 8B to 14B models is worth paying for, and 32B models are another clear step for code and complex instructions.

If the budget forces a choice, prioritize memory over speed. A card that holds the whole model will usually outrun a faster card that has to spill layers into system RAM.

Common Mistakes

Buying 8GB of VRAM. The RTX 4060 (8GB, now a previous-generation card) and similar cards cap you at 8B models at Q4_K_M. You will outgrow that quickly; aim for 12GB or more.

Ignoring power costs. An RTX 3090 box averaging 200W costs about $26 a month, or about $630 over two years, on top of the hardware. Factor that in when comparing with cloud APIs or a lower-power card.

Buying a second GPU instead of a bigger one. llama.cpp and Ollama can split a model across cards, but two 12GB cards bring two power budgets, two slots and more setup than one 24GB card. Our best GPU for local LLMs guide’s rule is to choose the smallest power envelope that gives you enough VRAM for the models you actually run.

Over-speccing the CPU and RAM. An expensive CPU adds little to inference speed when the model fits in VRAM. Put that money toward memory on the GPU instead; a modest used quad-core is plenty.

Skipping the UPS. GPU boxes draw a lot of power and do not like sudden shutoffs. A CP1500PFCLCD-class UPS gives the box time to shut down cleanly.

Starting with the Tesla P40. The P40 is a fine second card for someone comfortable with Linux drivers, forced-air cooling and headless operation. It is a poor first AI build for someone who just wants to run Ollama.

Budget Builds at a Glance

Used GPU prices swing too much to quote, so these builds list parts rather than totals. The GPU is the largest line item in each.

The 12GB Starter

• An existing or used desktop with a free PCIe x16 slot, 16GB of RAM and an SSD
• A used RTX 3060 12GB (compare completed sales)
• A 550W power supply if the desktop’s is smaller

Runs 8B models at up to Q8_0 and 14B models at Q4_K_M. It is enough for private, everyday chat and summaries.

The 24GB Build

• A desktop or budget build with a 750W power supply, 32GB of RAM and case room for a 3-slot card
• A used RTX 3090 24GB (compare completed sales and buy with a return window)

Runs 32B models at Q4_K_M with about 8K of context. In our judgment this is where local models become useful for code generation and analysis, though they still trail the best hosted frontier models.

Add Power Protection

• A CyberPower CP1500PFCLCD (1500VA/1000W, sine wave) for either build

With the UPS in place, you have a complete home AI inference server.

Wrap-Up

The minimum useful AI home lab is a used desktop, a GPU with 12GB or more of VRAM, and Ollama. If you already own a Radeon 780M mini PC or an Apple Silicon Mac, test small models there first; it may be enough.

If your budget covers a used RTX 3090, its 24GB of VRAM lets you run 32B models. If you want 70B-class models on one quiet box, look at unified-memory machines and budget accordingly.

Start with how much VRAM you actually need, pick a tier, and build around it. For a broader home lab that covers networking, storage and compute beyond AI, see the home lab starter guide.