Hi all, if you’ve not heard, there’s been a Caveman compression version of Qwen 3.6-27b and Qwen 3.6-35b-3ab.

https://huggingface.co/ProCreations/grug-27b-gguf

https://huggingface.co/ProCreations/grug-35b-v2

https://huggingface.co/ProCreations/grug-35b-v2-gguf

I’ve been playing around with 35B and I am able to run it on my ancient Quadro P1000 4gb at ~10tok/s.

But beyond that, the quality of the reasoning and the token discipline / output is actually higher than a stock, in my opinion.

You can read the benchmarks above; I am also uploading a HTML file here for your consideration.

(Sorry for the Limewire link; I dunno where else to share throw-away files. It’s HTML)

Anyway…I’m doing more testing right now…but so far, this is a good cook.

Or -

Grug good. Me like.

  • trem@lemmy.blahaj.zoneEnglish
    1·
    9 days ago

    Huh, does caveman just mean that it uses fewer fill words and less prosa, or does the output only look like that by chance?

    I thought, I read at some point that the caveman models use some pseudo-language for their thinking tokens. If they waffle less, that would be much more interesting to me…

    • SuspiciousCarrot78@aussie.zoneOPEnglish
      2·
      8 days ago

      Both; it has internally reduced thinking tokens AND it’s less verbose, so you get a boost on both prefill and print. I’ve benchmarked identical prompts and tool calls with stock Qwen 35B and Grug 35B; on average, what takes stock 1000 token/s, Grug does in about 200.