Hi all, if you’ve not heard, there’s been a Caveman compression version of Qwen 3.6-27b and Qwen 3.6-35b-3ab.

https://huggingface.co/ProCreations/grug-27b-gguf

https://huggingface.co/ProCreations/grug-35b-v2

https://huggingface.co/ProCreations/grug-35b-v2-gguf

I’ve been playing around with 35B and I am able to run it on my ancient Quadro P1000 4gb at ~10tok/s.

But beyond that, the quality of the reasoning and the token discipline / output is actually higher than a stock, in my opinion.

You can read the benchmarks above; I am also uploading a HTML file here for your consideration.

(Sorry for the Limewire link; I dunno where else to share throw-away files. It’s HTML)

Anyway…I’m doing more testing right now…but so far, this is a good cook.

Or -

Grug good. Me like.

  • brucethemoose@lemmy.worldEnglish
    2·
    9 days ago

    Be aware, q4_0 KV quantization really borks models.

    But the K is far more sensitive than the V. Try q5_1 for the K while leaving the V at q4_0 or q4_1; vram usage will be almost the same, but it should work dramatically better.

    As for throttling, try disabling turbo on the 8700.

    It doesn’t actually need turbo clocks for these models. The t/s loss I get from doing that on my rig is very modest.

    And like others suggested, try the QAT release. You might try the ik_llama.cpp for while you’re at it, at it should be faster with MoEs like this.