Hi all, if you’ve not heard, there’s been a Caveman compression version of Qwen 3.6-27b and Qwen 3.6-35b-3ab.

https://huggingface.co/ProCreations/grug-27b-gguf

https://huggingface.co/ProCreations/grug-35b-v2

https://huggingface.co/ProCreations/grug-35b-v2-gguf

I’ve been playing around with 35B and I am able to run it on my ancient Quadro P1000 4gb at ~10tok/s.

But beyond that, the quality of the reasoning and the token discipline / output is actually higher than a stock, in my opinion.

You can read the benchmarks above; I am also uploading a HTML file here for your consideration.

(Sorry for the Limewire link; I dunno where else to share throw-away files. It’s HTML)

Anyway…I’m doing more testing right now…but so far, this is a good cook.

Or -

Grug good. Me like.

  • SuspiciousCarrot78@aussie.zoneOPEnglish
    1·
    5 days ago

    Still fine tuning this. I’m finding that Grug has a particular affinity for larger -b and -ub sizes. I’m now able to hit 90-110 tok/s prefil (upto 155 tok/s fresh). Specific details below

    • 155.42 tok/s fresh 2,009-token prefill
    • roughly 90–110 tok/s on larger incremental/agentic prefills
    • roughly 11.3–12.9 tok/s sustained generation
    • 64°C peak CPU temperature
    • approximately 33 W peak CPU package power
    • no thermal throttling
    • ub and -b both 2048

    All this on a Quadro P1000 4GB card.

    In-sane.

        -m "%MODEL_PATH%" ^
    -t 8 ^
    -tb 8 ^
    -ngl 99 ^
    --n-cpu-moe 38 ^
    --flash-attn on ^
    --no-mmap ^
    --mlock ^
    -c 16384 ^
    -b 2048 ^
    -ub 2048 ^
    -np 1 ^
    --host 0.0.0.0 ^
    --port %PORT% ^
    --ui-mcp-proxy

    PS: Interestingly, dropping cache precision away from FP16 reduced tok/s generation by 18%. I don’t know why. I’m just fine tuning MTP now to see if I can eke out a few more tok/s, as MTP enabled grug-v2 just dropped

    https://huggingface.co/ProCreations/grug-35b-mtp-gguf

    PPS: Sadly also discovered my Tesla card has intermittent electrical fault. Took some troubleshooting to figure it out, but the long and short of it is you probably shouldn’t buy second-hand server cards off Ebay. Oh well, $100. On the upside, tweaking throttlestop, re-pasting and re-seating greatly improved thermals.