Dialing in local models on one GPU.
Dialing in local models on one GPU.
So I want a local model I can use as a daily coding assistant and I want it running as fast as this machine can possibly go.
Part of that is practical. Part of it is just for fun. I want to see how well these open models do on real work and how well they pair up with the big SOTA (state of the art) models I use every day. Could a local model handle the grunt work while the big one does the thinking? No idea yet!
Before I can answer any of that I need to know what the best setup even is.
Why I can't just run it
Ollama will happily load a model with its defaults and give you an answer. The defaults are not the best answer.
There are a TON of knobs and they all fight each other:
- How much of the model goes in VRAM vs system RAM
- How big the context window is
- What precision the KV cache (the model's memory of the conversation) uses
- CPU threads, batch size, mmap
Make the context bigger and the KV cache eats VRAM, so some of the model gets pushed out to system RAM and everything slows down. Shrink the KV cache precision and you get the VRAM back but maybe lose quality. Every model trades these differently.
On top of that my GPU is also driving my monitors. If a model eats all the VRAM the whole desktop crashes and takes everything down with it. Ask me how I know.
The machine
- AMD RX 7900 XTX with 24 GB of VRAM
- Ryzen 9 7900X (12 cores)
- 32 GB of system RAM
- Ollama on Linux with ROCm
The card is a glorious slab of silicon and the undisputed king of my desk.
Why MoE models
All the models I'm testing are MoE (mixture of experts) models. A normal model uses every weight for every token and that's a lot of weights. An MoE model has a big pile of "experts" and only wakes up a few of them per token.
So a 35 billion parameter model might only use about 3 billion per token. That means some experts can live in system RAM and it hurts way less than you'd expect. Most of them are sitting around doing nothing at any given moment anyway.
flowchart LR
subgraph GPU ["GPU - 24 GB VRAM"]
A["Attention layers"]
K["KV cache (the context)"]
E1["Some experts"]
R["3 GB kept free for the desktop"]
end
subgraph RAM ["System RAM - 32 GB"]
E2["The rest of the experts"]
end
A --> E1
A -.-> E2The experiment
Five models:
qwen3.6:35b-a3blaguna-xs.2glm-4.7-flashgemma4:26b-a4b-it-q8_0qwen3.6:35b-a3b-q8_0(39 GB, which very much doesn't fit)
For each one a benchmark script I built with Claude Code changes one knob at a time and records prompt speed and generation speed and memory use. Context goes from 8K up to 256K tokens. Then it takes the winners and tries them together at every context size, because the best setting at 8K is not always the best at 128K.
After that come quality checks with HumanEval, GSM8K and a needle-in-a-haystack test. A fast model that can't code is useless to me.
The 3 GB desktop reserve is enforced the whole time. Any config that can't keep it gets killed and marked as rejected.
It's already paying off
Some of what it found so far:
- On Qwen 3.6, moving experts into RAM on purpose beat Ollama's automatic split by 1.3x to 3x
- Using all 24 CPU threads instead of the 12 real cores took GLM from 49.6 tokens per second down to 8.6
- Gemma Q8 was crawling at 3.9 tokens per second until I turned mmap off, and then it did 37.6!
None of that is something I would have guessed. The automatic split is NOT the best split!
What I get out of it
For every model the script writes out a recommended config. It's a table of the best setup for each context size plus a drop-in file I can paste straight into Ollama:
[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_GPU_OVERHEAD=3221225472"
Environment="LLAMA_ARG_N_CPU_MOE=10"That's the real point of all this. Down the road I don't have to guess. If I want a fast coding model I know which one to load and with what settings. If I need a huge context for a big codebase I know exactly how far each model can go before it falls off a cliff or crashes my desktop. And when I start pairing these with the SOTA models I'm starting from the best this machine can do.
Qwen Q8 is up next with about 15 GB of it hanging out in system RAM and I have no idea if that's going to be awesome or a slideshow. Then the real world experiments start...