92
If open weight models are the future, U.S. AI companies are going to have a hard time
(www.fastcompany.com)
this post was submitted on 26 Jul 2026
92 points (98.9% liked)
Technology
86628 readers
3665 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I just wish I could buy enough memory to run one of these models locally. Specially Kimi K3
The other problem is bandwidth. Even if you can run Kimi K3 locally, AI usage patterns tend to be very bursty. Having the scalability of cloud compute to run multiple agents and sessions at once makes it much more usable.
My current workflow is passing a human written spec to an agent to implement with strict coding guidelines, architectural decisions, etc. The agent isn't making any decisions about the abstractions to use, it's just creating the objects and test suites. So I don't mind the slower bandwidth because I'm running the heavy agentic lifting over night with no need for human supervision.
But I fully appreciate that my workflow isn't the norm. In fact my workflow it's the exact opposite the AI grifters like Sam Altman are selling because it still involves a human with knowledge of the systems making different decisions.
If you're running stuff overnight on a homelab, you can get away with using very large models as agents in CPU RAM.
Another option is a hybrid approach: farm "dumb" privacy-insensitive calls to an API you can subscribe to for like $40 a year, and let your local LLM do the orchestrating. This is what I do: I run MiMo locally, but also farm calls out to the exact same LLM in the cloud when appropriate... And I don't mind one bit. I'm happy to tip them for open sourcing it in the first place.
How much RAM do you have?
I can run MiMo 2.5 at about 9 tokens/sec, on 128GB RAM, a 7800 and a 3090 in an SFF rig. That's a fantastic 310B model. I'm requantizing it right now, to see if I can speed it up with Dflash.
Still fantastic models can be run on 64GB or 32GB CPU RAM, as long as you have some GPU. We're awash in sparse models these days.
Same, getting ~3 trillion parameters in consumer hardware is rough.
If Nvidia has any foresight they'll see the writing on the wall and start getting higher memory Spark style SMB inference machines, few people in the long run are going to pay retail API token costs,
Especially giving the lack of trust organizations rightfully have in big AI companies