• GenderNeutralBroEnglish
    8·
    17 hours ago

    If you’re counting models that can be run locally but are too big to run on consumer hardware, then it’s pretty small. Like, they’re only lagging by a few months. But you you’d need like $40k of hardware to run them effectively.

    If you mean just the models that you can run on consumer hardware, like, say qwen3.8-27b, then it’s more like 1-2 years behind. You can compensate for the lack of raw capability with a good harness, agentic loops, and human effort. As much as AI has a reputation for being low-effort, it actually requires a lot of skill to use very effectively.

    • Jo Miran@lemmy.mlEnglish
      2·
      12 hours ago

      I guess the question is, what gets priority, RAM or GPU. I can get a dual 192GB 6000 nvidia with 256GB of RAM for $50k, but that still doesn’t get me the capability of top models. Or I could get a 1TB of RAM tower server with maybe one 96GB nvidia 6000 for about the same. Does the RAM capacity mean I can fit the larger model but it’s just going to be slow?

      • GenderNeutralBroEnglish
        1·
        3 hours ago

        That’s a good question and it’s hard to say for sure. There’ve been some advancements this year in offloading, using both VRAM and system RAM together. With MoE models and predictive loading you can get pretty good results that way…sometimes. I’ve never tried it myself, though I really should because my 24GB GPU is a serious bottleneck.

        You could also think about getting a 512GB Mac Studio (coming soon). Not as fast as a GPU, but maybe faster than a GPU/DDR5 mix? And definitely cheaper per GB.

        I realize now that the truly biggest models need way more than $40k of hardware. The biggest Qwen3.8 is 2.4 TRILLION parameters. Yikes.

      • BeefAndPoultry@lemmus.orgEnglish
        3·
        10 hours ago

        With Strata, things have gotten more flexible. Similar optimizations are coming to llama.cpp eventually.

        https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#pick-by-memory

        It says Qwen 3.8 Flash Next can run with just 8GB VRAM and 96GB RAM for a good quant of it. This is a really good model, and the new version 4 should be coming in the next month or so, probably the same size but smarter.

        DeepSeek v4.1 Flash is probably even smarter, says it needs 12+GB VRAM and 64GB of RAM. For decent speed I think you’ll want at least 32GB VRAM and 256GB RAM for this one. 96GB VRAM should be very good.

        You’re gonna have big diminishing returns chasing for even bigger models, partly because Strata doesn’t support anything bigger and llama.cpp is currently not nearly as optimized. There might be an inference engine I don’t know about for the biggest models.

    • grue@lemmy.worldEnglish
      5·
      16 hours ago

      If I’ve got 128GB of RAM but it’s really slow RAM (DDR3), can I run a good model really slow or will it just not work at all?

      • GenderNeutralBroEnglish
        3·
        15 hours ago

        It’ll be slow but it’ll run.

        Though even 128GB isn’t enough for the biggest models. 512 would open up more options and the very biggest need like 2-4TB, totally out of scope for personal use.

        You could try a medium-big MoE model (Mixture of Experts) with less than 100b total parameters. Maybe start with something modest like Qwen3.6-35B-A3B at 8bit quants. That’ll leave you with plenty of memory left over for context size.