WEBVTT

NOTE
Subtitles for the Run AI on your own computer walkthrough. They match the captions
drawn on the video, and there is no spoken narration. The music is an original
score. Cue times follow the film timeline in seconds.

1
00:00:00.400 --> 00:00:04.400
Can a capable language model run on hardware you already own?

2
00:00:04.600 --> 00:00:08.400
The promise: your prompts stay on your machine, and you choose the model.

3
00:00:08.600 --> 00:00:11.600
Our job: pick a model that fits, install Ollama, and run Olmo 3.

4
00:00:12.400 --> 00:00:16.000
Most AI runs on a provider’s servers. A local model runs on your own computer.

5
00:00:16.200 --> 00:00:20.200
Your prompt stays on your machine, and it works offline once the download is done.

6
00:00:20.400 --> 00:00:24.000
You choose the version and keep the files, so nobody can retire the model on you.

7
00:00:24.200 --> 00:00:27.800
The trade-off: the biggest models are hosted, and local models are limited by your memory.

8
00:00:28.000 --> 00:00:31.600
And local is not automatically private. Check cloud features, network settings, and where the model came from.

9
00:00:32.400 --> 00:00:35.800
A local model has three layers. First, the weights: billions of numbers saved in one file.

10
00:00:36.000 --> 00:00:39.400
Olmo 3 7B is about 4.5 gigabytes, and the 32B version is about 19.

11
00:00:39.600 --> 00:00:43.000
Second, the runtime: the program that loads the weights and does the math. Ollama builds on engines such as llama.cpp.

12
00:00:43.200 --> 00:00:46.600
Third, the hardware. Memory holds the model, and memory bandwidth sets how fast words appear.

13
00:00:46.800 --> 00:00:50.400
So memory is roughly parameters times bits per weight, divided by eight, plus context and overhead.

14
00:00:50.600 --> 00:00:53.600
Quantization stores each weight in fewer bits. Four-bit makes the file about four times smaller than sixteen.

15
00:00:54.400 --> 00:00:58.000
In the planner, watch what limits a local model.

16
00:00:58.200 --> 00:01:02.200
Memory is the limit, not just compute. Bandwidth sets tokens per second. Context length costs memory.

17
00:01:02.400 --> 00:01:05.600
With Ollama, you run, pull, check with ps, and stop a model.

18
00:01:05.800 --> 00:01:09.600
Predict: can a laptop with 16 GB of memory run Olmo 3 32B Think? What about 7B Instruct?

19
00:01:10.400 --> 00:01:14.000
This is the planner from the lesson. Choose your hardware and a model.

20
00:01:14.200 --> 00:01:18.400
Start with an Apple Silicon Mac that has 16 GB of unified memory.

21
00:01:18.600 --> 00:01:22.200
Pick Olmo 3 7B Instruct, at 4-bit, with the default context of 4,096 tokens.

22
00:01:22.400 --> 00:01:26.600
The weights take about 4.4 GB, the context cache about 2.1, and overhead about 1. That is 7.6 GB in all.

23
00:01:26.800 --> 00:01:30.600
The GPU can use about 11 GB of the 16, so it fits, and it runs at around 16 tokens per second.

24
00:01:30.800 --> 00:01:34.400
Now switch to Olmo 3 32B Think.

25
00:01:34.600 --> 00:01:39.200
The weights alone are about 19 GB. With cache and overhead, it needs about 21.5 GB.

26
00:01:39.400 --> 00:01:43.800
That is far more than this laptop can use, so the planner says: too big for this machine.

27
00:01:44.000 --> 00:01:48.400
It also suggests a fix: Olmo 3 7B Instruct fits this machine.

28
00:01:49.000 --> 00:01:53.200
Now try a PC with a GPU: 24 GB of video memory, and a high-end card.

29
00:01:53.400 --> 00:01:57.800
Same 32B Think model, same 21.5 GB. This time about 23.5 GB of video memory is available.

30
00:01:58.000 --> 00:02:02.800
It fits, but tight: close other apps or shorten the context to stay safe.

31
00:02:03.000 --> 00:02:07.600
Fast video memory makes it quick: around 31 tokens per second.

32
00:02:07.800 --> 00:02:12.600
Memory decides whether a model runs. Memory bandwidth decides how fast.

33
00:02:12.800 --> 00:02:19.600
Now, back to our prediction.

34
00:02:20.400 --> 00:02:24.200
So, can a 16 GB laptop run Olmo 3 32B Think? No. It needs about 21.5 GB.

35
00:02:24.400 --> 00:02:28.000
Can it run 7B Instruct? Yes: about 7.6 GB, at a comfortable pace.

36
00:02:28.200 --> 00:02:31.600
The method works for any model: compare what it needs with what you have.

37
00:02:32.400 --> 00:02:36.200
Step one: install Ollama. On macOS and Windows, download the app. On Linux, run the install script.

38
00:02:36.400 --> 00:02:40.400
Step two: run a model. The first run downloads it, then you can chat. Type slash bye to leave.

39
00:02:40.600 --> 00:02:44.400
Step three: check where it landed. ollama ps shows whether it is on your GPU, your CPU, or split.

40
00:02:44.600 --> 00:02:49.000
Olmo 3 7B Instruct and 7B Think are each about 4.5 GB. Instruct answers directly. Think shows its reasoning.

41
00:02:49.200 --> 00:02:53.400
Olmo 3 32B Think is about 19 GB, for the hardest problems, and needs 24 GB or more.

42
00:02:53.600 --> 00:02:57.600
Olmo is fully open: the weights, the training data and the code are all public.

43
00:02:58.400 --> 00:03:02.400
Ollama listens on your machine only, at 127.0.0.1, port 11434, unless you change it.

44
00:03:02.600 --> 00:03:06.800
Call it with curl, or from Python with the ollama library.

45
00:03:07.000 --> 00:03:11.000
There is also an OpenAI-compatible endpoint, so many existing tools work unchanged.

46
00:03:11.200 --> 00:03:15.200
The default context is 4,096 tokens. Set OLLAMA_CONTEXT_LENGTH for more, which costs more memory.

47
00:03:15.400 --> 00:03:18.800
Models stay loaded for five minutes. Run ollama stop to free memory now.

48
00:03:19.000 --> 00:03:21.600
And set OLLAMA_NO_CLOUD to 1 to stay local-only.

49
00:03:22.400 --> 00:03:25.800
To compare your options, start with the hardware.

50
00:03:26.000 --> 00:03:30.000
Apple Silicon shares one pool of memory. A GPU is fastest when the model fits in video memory.

51
00:03:30.200 --> 00:03:33.400
CPU only works almost anywhere, but it is slower, so keep to small models.

52
00:03:33.600 --> 00:03:37.600
Runtimes: Ollama is the easy path. llama.cpp is the engine under many tools, with the most control.

53
00:03:37.800 --> 00:03:41.400
LM Studio is a free desktop app. vLLM serves many users on GPUs.

54
00:03:41.600 --> 00:03:45.600
Model families run from fully open to custom terms, so read each model card.

55
00:03:46.400 --> 00:03:49.800
Here are eight questions for choosing and running any local model.

56
00:03:50.000 --> 00:03:54.000
How much memory can it really use? Do the weights fit? Parameters times bits, divided by eight.

57
00:03:54.200 --> 00:03:58.000
How much context do I need, and what does it cost? Dense or mixture-of-experts: memory versus speed?

58
00:03:58.200 --> 00:04:02.200
Is the license fully open, open weights, or restricted? Does my runtime support the architecture?

59
00:04:02.400 --> 00:04:06.400
Is anything leaving the machine: cloud features, the network, telemetry?

60
00:04:06.600 --> 00:04:09.600
And how will I check the answers? Models make mistakes.

61
00:04:10.400 --> 00:04:13.400
Two questions for the room.

62
00:04:13.600 --> 00:04:19.400
Which of your tasks could use a model that never leaves your device, and which still need a bigger hosted one?

63
00:04:19.600 --> 00:04:25.400
What do you gain when a model’s training data and code are public too, not only its weights?

64
00:04:26.400 --> 00:04:30.400
That is the lesson: pick a model that fits, install Ollama, and run it.

65
00:04:30.600 --> 00:04:35.000
Your prompts stay on your machine, and you choose the model.

66
00:04:35.200 --> 00:04:39.400
Memory is the limit. Do the math first, then run ollama ps to check.