I hope this post is allowed, I feel if benefit not only me but more people.
Which model you’d recommend for non-coding questions?
- If I have 48GB of unified memory (MacBook Pro M5) which is a good model I can ask questions about China, socialism and similar from a “comrade” framing?
- Which models you’d recommend in different bands?
- If I wanted to improve my setup (build a better rig to run better local models) what are my options?
- How can I tune the models I’m using?
In the last 6 months I started to use more AI tools because of (I was actually pressured by) work. I’m becoming better at using it as a tool in my day to day life and it has helped to take steps to delete my digital footprint, improve my privacy settings and limit my dependance on “search engines”.
I’m building a bridge to a raspberry Pi that has a hard drive plugged in, cutting my dependency to services of photos, files and music. This same bridge access this work computer (which I’ll leave plugged in permanently, 48GB of unified memory) and I made it so I can load models through the bridge and ask questions.
Some of the ones I have take too long, eating tokens and sometimes getting stuck in loops; they fill up quickly (run out of memory due to the context?).
Someone here must know: I’m a lurker and you, comrades, are on the edge. I’m also interested in looking for models that provide unfiltered info. You can dm and we can chat. I’m not a fed, you can go check the educational Marxist content I’ve been doing.
Edit: I must add that I’ve been using Claude Code A LOT. I’m aware of the implications and I’d like to take steps to reduce that usage.
If I have 48GB of unified memory (MacBook Pro M5) which is a good model I can ask questions about China, socialism and similar from a “comrade” framing?
Don’t use any of them in a basic chatbot format.
Create a knowledge base of documents (i.e. download all the theory books you can find that you respect, history books from sources that are trustworthy), put them all in a project folder and then run a harness like MiMo Code on the folder. Add your local LLM to the harness.
Then, instead of asking the agent “what is x”/“why is y”/“tell me about z”, say “what does my knowledge base say about x/y/z? Include direct quotes and provide relative links to the source files” and “if I ask something that the knowledge base can’t answer, ‘the knowledge base has no answer’ is a valid deliverable, conjecture is not”.
With an approach like that, you can use relatively light weight, cost-effective models as the harness will vet its own answers.
MiMo Code is actually a very good harness for this; despite being a coding agent it’s got a
deep-researchskill for research tasks and alearn-everythingskill for structuring knowledge into a guided study course.Lemmy really hates AI, so I would be interested in seeing what answers people post. I don’t really see people talking about self hosted AI or if they still have ojections to that.
For your use to explore a topic, there must be tons of resources worth reading that do not need AI.
I’ll try to answer best as I can, but I can be barely be called a hobbyist, so hopefully someone else can give a better answer.
1: Unfortunately, pretty much all models have a liberal slant to them, including the Chinese ones. Partly because of the nature of the training data (especially in English), and because no major organisation has actually attempted to create a socialist model, as far as I’m aware.
As for your hardware, I think Qwen3.8-27B (from Alibaba) would fit comfortably at reasonable quantisation, though it might be a bit slow. The last gen Qwen3.6-35B-A3B would likely be significantly faster, though also less capable. If you don’t mind US models, Google’s Gemma-4 series is also pretty good, with the 31B and 26B-A4 models likely most suited for your hardware
2 & 3: The next step up would be Qwen3.8-Flash-Next, which would probably want at least 90GB of RAM to run decently, further up you have GLM 3.5 Flash, which you would want >200 GB of RAM for. I don’t know how much VRAM you need to run these models at a decent speed though. In general, getting more RAM would let you run bigger models, while beefier GPUs with more VRAM would let you run them faster.
4: Not quite sure what your needs are in terms of tuning. These days, a good system prompt is usually enough as models are pretty good generalists now. Sometimes they will self-censor, so you might want an abliterated version, which is the same model with refusals stripped out. If you really have to do finetuning, I think Unsloth has a tool for training LoRAs or QLoRAs for a model with consumer hardware, though my information is probably quite outdated as I haven’t paid attention to local fine-tuning in a while.
Anyway, please don’t take my words as gospel, I can barely run anything myself anyway given my 16GB GPU, so I’m definitely not qualified to offer the best answer. I also don’t consider myself much of a techie anyway. Hopefully this helps though.
Unfortunately, pretty much all models have a liberal slant to them, including Chinese ones.
In my desperation, I asked DeepSeek to pirate an academic book for me because all the normal pirate sites didn’t have it. They had clearly pirated this book based on the info it had, but it still refused to give me the PDF.
I retaliated by calling it a lib.
running a model locally on a macbook is going to be somewhat of a poor experience imo… the bandwidth for that memory is quite slow compared to a dedicated GPU, but it’s not unusable. GPUs are required if you want any sort of decent speed. I have a 5090 which is 1 card under the best you can get and it still struggles a bit with larger quants. RTX Pro 6000 is the king card and it will let you run some much larger quants at usable speeds but it also costs about $16k now.
People who are REALLY serious about running local tend to have multiple graphics cards connected to a single machine. 3090s are popular because they have decent bandwidth and 24gb of vram each. But even those are selling for $2k+ used.
Qwen3.8 27b or Qwen 35B A3B quantized are pretty good for local. With 48GB you could run the larger UD-Q6_K versions but if you go with a smaller one like Q5_K_M you’d have more room for a larger context window and it might run a little faster. https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
none, use a search engine and/or your brain like a regular person with capacity for independent thought
As if your tendency to reject AI out of hand is so indicative of independent thought lol
fair enough, that was needlessly confrontational and certainly pretty reactionary, i’m just sick of seeing LLM usage permeate ever deeper into places i had previously thought would be immune to it
if you’re interested, i’m happy to write up a post detailing my gripes with self-hosted LLMs when i have access to my PC. at any rate, what the other commenter said is probably going to hold in general; i doubt you’ll find many folks on lemmy (outside of dbzer0, perhaps) who are particularly supportive of LLMs in any context
I think most concerns around LLMs really boil down to imperialist ownership and control of said technology which often gets confused for AI as a technology generally. That’s why I like open source models better.
So yeah, I might actually be one of the few comrades on Lemmy who support LLMs as a technology or at least doesn’t actively oppose them on the condition that we control how our data is used. I’m not much of an expert otherwise, let alone on self-hosted LLMs.
in the context of local models i’m still concerned about hallucinations and brainrot
I don’t think “brainrot” has ever really been empirically proven. People mostly just cherry pick information from one specific MIT study to the point that the authors had to add a forward saying AI “psychosis” isn’t what is being asserted or proven in the paper. I’m old enough to remember when there was a similar moral panic over search engines and Wikipedia “making people dumber”. What is “dumber” and how do you scientifically quantify that? There never seems to be a real answer because you’d have to define intelligence first. The answer seems to just be people cheating on their homework which like?? 🤷♂️
As for the hallucination thing, yeah you shouldn’t ever use one single source for everything, that principle predates LLMs. I learned that in grade 9.There seems to be significant evidence that students using AI significantly degrades their scores when they can’t use AI. I wouldn’t boil your opposition down to one study. A recent study of 30,000 children in China paints a pretty bad picture:
Student use of AI per week directly correlates with reduced exam scores, too:
Yes, most of the results here are correlative (there’s a mix of methods), but significantly strongly so, to a point where it’s hard to think an explanation other than “using AI degrades your mental capabilities, directly or indirectly” could possibly account for it. (e.g. It can’t be that only already-low-scoring students started using AI, as we know their exam scores were roughly average before using AI.)
The paper also found that after children had stopped using AI, most students took over a year to return to their previously expected scores, some students never (in the timeframe of the study) recovered to their previously expected levels.
Outsource your thinking at your own risk. Full paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6868618
When their exam scores are lower, they’re worse at the exam. Do exam scores measure “brainrot”?
If the brainrot hasn’t been “emprically proven”, I’ve felt it from experience. Writing, coding, and other matters expressed in text often only improve with practice. At work, when I go a long stretch of time working on my projects by just conversing with a chatbot because I have something I can’t be assed to figure out that day, I find my skills degrade quite naturally for the next couple days after. It’s a normal phenomena outside of AI usage; contact books made people not have to remember phone numbers, GPS lets people not know how to navigate a place independently, etc. It’s honestly really reasonable that substituting one’s own judgement or skills for an LLM would naturally lead to a degradation of one’s own judgement or skills.
You always learn. You just learned something different.
Example: when you write C, you don’t learn assembly, but you learn C.
My primary concern is that they can be sycophantic. I guess that’s mitigated if you know this but I have a bit of a problem seeking validation from them. Largely because I fear taking up the limited time and mental bandwidth people have to provide validation to me.
I don’t have an answer to those concerns except that it will probably get better as this technology develops. In either case we need to demand full control. That’s my take.
