I’ve been back’n’forth for a while whether to invest in a second 5060Ti to be able to go dual GPU on my main workstation. The target has always been to be able to run Qwen 3.6 27B at a usable model quant, kv-cache quant and context size.
Today I tried out llama.cpp with the rpc option. I sort of knew at the back of my head that there were projects doing cross-computer inference but I didn’t know it was built into llama.cpp.
Sure, this requires that you have another machine on your local network (latency matters) with a GPU - but in my case I actually have two. One server that has a 12GB A2000 and one Bazzite box with an AMD something also with 12GB VRAM.
I just compiled the latest llama.cpp HEAD with the rpc option enabled on my headless server and launched my main llama-server pointing to it. With the 12GB of the A2000 in the mix I’m now able to comfortably run the 27B model Q4_K_M, Q8_0 kv-cache and 140000 context size, and I’ve still not maxed out the available VRAM.
PP at ~400 (a bit low, will see if that can be tuned*) and TG at 25 (MTP and ngram enabled) makes this perfectly usable. That’s the token/second during inference I would’ve expected from a dual GPU setup in the workstation anyway.
So, if you also happen to have another machine on your network with spare VRAM - try out llama rpc. I have no idea why it took this long for me to do.
*) This is literally my first run with random defaults - tuning starts now
Last time I ran rpc-server, it kept crashing on me badly. I’ve seen some updates here and there in the release notes, but are they actively keeping it working now? I don’t want to switch my tooling over again for something that isn’t rock solid.
Surely we should have a hardware solution soon, that’s just a bunch of vram? I saw an article recently where a dude got a server GPU or something and jimmied it up to run in his PC. Worked great apparently
Yeah Nvidia V100’s and AMD MI50’s or V620’s are the accessible ones
May want to also check out Project Ray: https://github.com/ray-project/ray
Sits on top of vLLM. What the big guys use.
The problem I have with vLLM is that it’s built for the big guys as you put it, which means you need the hardware the big guys have or you have to start making trade offs. Llama.cpp was built first as a CPU inference platform. They added in GPU offloading later, that’s kind of why some of the flags names can seem backwards.
Interesting. What kind of network do you have between the two machines?
Regular 1Gbit/s with three switches in total between them - distance about 40m (different buildings).
