llama_moe_cache: run missed experts on CPU, add tuning script by am17an · Pull Request #30966 · ggml-org/llama.cpp
github.com
Overview Depending on --moe-cache-cpu-share, this PR executes experts on the CPU. Depending on how fast the CPU can read experts, this can affect decode speeds a bit. Since --moe-cache-mib and --m...

am17an: Before you guys bash llama.cpp again - I know it’s not there yet. I’m working on it.

am17an: I’m working towards an “auto” mode here, thinking about a good UX.

I tried this on my Ryzen 2600, 32GB DDR4 RAM, RTX 2080 8GB, PCIe 3 desktop. The tuning script takes about 5 minutes to run but it gives you recommended settings. I tested Tiel 35b, Gemma 26b, and LFM 2.5 24b, all without mmproj (or with --no-mmproj-offload). If you have PCIe 5, your numbers will probably look different even with the same 8GB VRAM.

My real world numbers seem about 20% slower than what the tune script is predicting, but still I’m getting the fastest speeds I’ve ever seen in llama.cpp on my computer, and I’ve tried a few of these different attempts at MoE caching.

Tiel 35b

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 35.3 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 50.1 t/s
4/6 3136 MiB, share 0: 41.1 t/s
5/6 3136 MiB, share 0.5: 47.3 t/s
6/6 3136 MiB, share 0.95: 53.5 t/s

[The script pauses here for a few minutes...]

Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7284 MiB free, cache up to 3188 MiB

  cache MiB  share   t/s  hits
       1856      0  35.3   73%
       1856    0.5  38.6   66%
       1856   0.95  50.1   64%
       3136      0  41.1   81%
       3136    0.5  47.3   79%
       3136   0.95  53.5   73%

machine: uploads 10.9 GiB/s, CPU 40.8 GiB/s, CPU part 0.00 ms, overlap 1.4 ms/GiB, rest 14.14 ms per token, fit within 0.7%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  25.3  32.8  39.6  42.4  43.9  44.6
       1856  32.8  39.8  46.3  48.8  49.9  50.3
       3136  41.6  47.7  52.9  54.7  55.4  55.4

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.8 (52.9 t/s, 41.6 with share 0)

Interesting to note that 1856MB and 0.95 share is almost as fast as 3136MB and 0.8 share. This means it might be worth reducing the cache size in order to use MTP, or maybe I should start using a higher quant.

But looking at the “predicted” table, it seems like the fastest would actually be 3136 MB and 0.98 share for 55.4 tok/sec? Not sure why it doesn’t recommend that instead, maybe it’s an expected peak not average.

Forcing Lower VRAM Usage

You can use --max-mib to force it to tune your settings with a lower VRAM usage, just make sure you put this argument before the llama-server portion of the arguments. It seems to work in steps, so not every number is respected, and the initial data collection run still seems able to use more than your requested amount, but the recommendation will respect it.

$ python3 scripts/moe-cache-tune.py --max-mib 2500 -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 34.1 t/s
2/6 1856 MiB, share 0.5: 37.4 t/s
3/6 1856 MiB, share 0.95: 49.8 t/s
4/6 3136 MiB, share 0: 41.6 t/s
5/6 3136 MiB, share 0.5: 44.9 t/s
6/6 3136 MiB, share 0.95: 54.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7292 MiB free, cache up to 2500 MiB

  cache MiB  share   t/s  hits
       1856      0  34.1   73%
       1856    0.5  37.4   66%
       1856   0.95  49.8   64%
       3136      0  41.6   81%
       3136    0.5  44.9   79%
       3136   0.95  54.1   73%

machine: uploads 8.7 GiB/s, CPU 47.0 GiB/s, CPU part 3.06 ms, overlap -12.2 ms/GiB, rest 11.50 ms per token, fit within 1.3%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  23.2  31.6  39.4  42.8  44.7  45.7
       1856  31.4  38.3  45.7  48.8  50.3  50.9
       2496  36.9  42.5  49.1  51.8  53.0  53.4

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (51.8 t/s, 36.9 with share 0)

It recommended a slightly higher “share” meaning it allows more to run on the CPU, to reduce movement of data.

Testing with GPU Power Limit

I used sudo nvidia-smi -pl 125 to limit my GPU’s power from 225 down to 125:

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 29.5 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 48.9 t/s
4/6 3136 MiB, share 0: 41.4 t/s
5/6 3136 MiB, share 0.5: 46.8 t/s
6/6 3136 MiB, share 0.95: 52.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7288 MiB free, cache up to 3192 MiB

  cache MiB  share   t/s  hits
       1856      0  29.5   73%
       1856    0.5  38.6   66%
       1856   0.95  48.9   64%
       3136      0  41.4   81%
       3136    0.5  46.8   79%
       3136   0.95  52.1   73%

machine: uploads 7.6 GiB/s, CPU 26.4 GiB/s, CPU part 0.00 ms, overlap 0.0 ms/GiB, rest 11.30 ms per token, fit within 7.9%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  21.1  28.8  36.1  39.3  41.0  41.7
       1856  29.0  37.0  44.9  48.0  49.4  49.8
       3136  39.4  47.5  54.6  57.2  58.1  57.8

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.9 (57.2 t/s, 39.4 with share 0)

You can see GPU power limits don’t affect token generation much, it has a bigger impact on prefill speed. The cpu share is a little higher because the power limit makes the expert movement slightly less valueable. The predictions seem a bit… unpredictable. This run had slightly slower figures for every test, but ended up predicting a higher speed, who knows why.

Gemma 4 26b

With Gemma 4 26b QAT I got:

recommended: --moe-cache-mib 3328 --moe-cache-cpu-share 0.8 (50.1 t/s, 40.3 with share 0)

And with 131072 context:

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (45.7 t/s, 30.5 with share 0)

LFM 2.5 24b

With LFM 2.5 24b QAD I got:

recommended: --moe-cache-mib 5056 --moe-cache-cpu-share 0.9 (75.7 t/s, 51.5 with share 0)

I wonder why this one suggested a high share when the cache size nearly half of the model size. Maybe it’s because the experts that aren’t cached are infrequently used. So when your cache is big enough to hold every frequently used expert, they aren’t worth moving back and forth.