1. Why run AI locally
It costs almost nothing per question. You already paid for the computer. Ask a local model a thousand questions and there is no API bill. Electricity is the one cost you still pay.
It is private by default. Contracts, medical notes and private journals stay on your machine. Nothing is sent to an AI company.
It always works. On a plane, while camping, or when your AI provider's API is down, a local model still runs.
You can choose the model. There are thousands of open models, including fine-tuned versions for specific jobs. Some of them answer questions that ChatGPT and Claude refuse.
Local models are good today at chat, writing, summarizing documents, code help, working with private data, and small always-on agents. Local AI today is roughly where paid cloud models were a year ago.
2. What your machine can run
A model is a large set of numbers stored in files. To run at a usable speed, most or all of those numbers need to fit in fast memory. If the model does not fit, it gets much slower or does not run at all.
- On a PC, the memory that counts is your Nvidia graphics card's VRAM. Regular system RAM is not the same thing.
- On an Apple Silicon Mac (M1 or newer), the CPU and GPU share one pool called unified memory, so your total memory is what counts.
- On an Intel Mac, local models are not a good fit.
Here is the rough map of what runs comfortably. The "B" means billions of parameters.
| Memory (VRAM or unified) | Model size that runs comfortably | What it feels like |
|---|---|---|
| 8 GB | 3B to 4B | Quick chat and simple tasks |
| 16 GB | 12B to 14B | Solid everyday helper |
| 32 GB | About 30B | Seriously capable, including coding |
| 64 GB and up | 70B class | Getting close to cloud quality |
3. Pick a runner
A runner is the app that loads the model and lets you talk to it. There are dozens. You only need to know three:
- Ollama: open source and the simplest to start with. This is the one I use.
- LM Studio: a desktop app with a friendly interface.
- llama.cpp: the engine most runners are built on, for people who want full control.
Most runners are wrappers around the same llama.cpp engine. On Apple Silicon there is also Apple's MLX engine, and Ollama and LM Studio both support it now. Pick one and move on.
4. Pick a model and read its name
A model name works like a license plate. Take qwen3-coder:14b-q4:
- qwen3 is the family, which tells you who made it.
- coder is what it is tuned for, in this case code.
- 14b is the size: 14 billion parameters.
- q4 is the compression level, called quantization.
Quantization shrinks a model so it fits in less memory, with a small loss in quality. For example, a model that is 28 GB at full size is about 12 GB at Q5 and about 9 GB at Q4. Many setups use Q4 because it is fast and the quality loss is usually small.
Dense vs. mixture of experts (MoE). A dense 30B model uses all 30 billion parameters for every word, so it is heavier and slower. An MoE model has the same total size but only part of it is active at a time. A name like 35b-a3b means 35 billion total, 3 billion active. An MoE model still needs the memory for its full size, but the active size tells you how fast it will feel.
Specific model picks change every month, so check what is new before you download. The course keeps a current shortlist for each memory size.
5. Run your first model
Install Ollama from ollama.com, then run a model from your terminal:
ollama run qwen3:8b
>>> How far is the moon from Earth?
The desktop download also gives you a chat window that looks and works like ChatGPT, with all your downloaded models in a menu.
One lesson I learned on camera: close the apps you are not using. A local model shares your computer's memory with everything else. While recording the course, a reply got much faster once I closed about 20 open apps.
6. Connect a local model to real work
Chatting is the start. The useful part is plugging a local model into the tools you already use. Ollama can launch coding agents like Claude Code and OpenAI's Codex on top of a local model:
ollama launch codex
In the course I asked Codex, running a local 35B mixture of experts model with 3 billion active parameters, to create a folder and build a simple website for a made-up coffee shop. It made the folder, wrote the HTML file, and had a working page in 1 minute 20 seconds. I did not copy or paste anything. That was on my 128 GB Apple Silicon M3 Mac.
You can also call a local model from your own scripts and small apps through its local API, the same way you would call a cloud API.
7. When local is the wrong choice
Use a cloud model when you need:
- The hardest reasoning, like a complex startup problem where you want the best answer available.
- Agent sessions that run for hours.
- Frontier-level coding, or tens of subagents running in parallel.
- The right answer every single time.
- Your computer for heavy apps at the same time, like Photoshop.
The grown-up setup is hybrid. Easy tasks stay local. Hard tasks go to the cloud. One pattern that works right now: a big cloud model like Claude Opus writes the plan, a local model does the coding, and the big model checks the result. You pay for the reasoning and get the routine work free.
8. The full course, still short
Why I made it a short primer
Most people do not need a 15-hour course to start with local AI. They need to know what their machine can run, which app to install, which model to download, and one real job to give it. That fits in about an hour.
Once a model is running on your own computer, you learn the rest by using it. The course grows from there: when students ask for a topic, I add a lesson.
This page is the short version. The course walks through every step on screen, with the commands in each lecture's resources. It also covers local vision, speech to text, text to speech and image generation.
Udemy course by Saurav Sharma · about an hour
Local AI Crash Course: Run Models Locally and Do Real Work
Get the course on Udemy → Discount codeLOCAL_AI_YT applied in the link
Want the free version first? The 26-minute YouTube primer covers the same four decisions.
Stuck on a setup, or deciding whether your team should run models locally? Book a 55-minute call.