this post was submitted on 26 Jul 2026
93 points (98.9% liked)

Technology

86628 readers
3665 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots


founded 3 years ago
MODERATORS
top 35 comments
sorted by: hot top controversial new old
[–] DJKJuicy@sh.itjust.works 9 points 1 hour ago (2 children)

I ran DeepSeek and Llama and Mistral at home on my consumer grade gaming PC.

With a little tweaking of the system prompts and configuring web search, I was running a local LLM that felt pretty darn close to the commercial LLMs.

With this technology out in the open internet where you can download the models in a few hours I don't see how the commercial AI companies are going to last. If selling "Artificial Intelligence" subscriptions is all your company does for revenue, you're screwed.

I downloaded and ran an LLM that I could have a conversation with and feed basic coding problems to for basically zero dollars and ran it on my puny gaming machine...puny compared to enterprise-class hardware. It would be trivial for a company with a very moderate budget to buy some servers and start running their own LLMs that they can use to feed all the PII and HIPPA data they want.

[–] stankmut@lemmy.world 4 points 47 minutes ago (1 children)

Since you mention using Ollama, you probably aren't running actual deepseek on your pc. Ollama took a Qwen model that was finetuned using deepseek output and named it deepseek.

Those are pretty out of date models at this point. Right now, the model most people would recommend for consumer gaming hardware is Qwen 3.6 27b.

[–] DJKJuicy@sh.itjust.works 2 points 36 minutes ago

I actually tried Qwen 3.6 27B but it wouldn't quite fit in my 6900XT so I had to go down to the 14B. I don't have the tools or the skillset to really test the capabilities of an LLM but with some very rudimentary system prompts it felt quite natural to me. Shockingly natural considering that talking to a real LLM running on my own PC felt like it was smarter than the Majel Barrett computer on Star Trek:TNG...

[–] brucethemoose@lemmy.world 5 points 1 hour ago (1 children)

And Llama and Mistral are ancient history at this point.

The cutting edge of local is lightyears better now. It's basically where ChatGPT/Anthropic were not that long ago, with a bit less world knowledge because of the size.

[–] DJKJuicy@sh.itjust.works 4 points 1 hour ago (1 children)

What's the cutting edge now? Skool me...I want to try it. Can I grab one using ollama?

[–] brucethemoose@lemmy.world 1 points 1 hour ago* (last edited 1 hour ago) (2 children)

https://sleepingrobots.com/dreams/stop-using-ollama/

And this is just the tip of the iceberg for ollama. They're the same kind of scammy tech bros as OpenAI.

The best setup depends on your hardware. There is no "easy button" unfortunately, quantized LLMs are just too intense and finicky to run without making some informed choices.

It also depends on what you want to do with the LLM. For example, some are too slow or bad at long context for agenic use, some quantizations are great at scripts but terrible outside that, or vice versa.

But LM Studio and Qwen 3.5 35B Q4 is probably the "easiest" flat recommendation I can make.

Or... honestly, just pay $40 for basically unlimited usage for a year from an API, then roll your own frontend.

[–] naught101@lemmy.world 2 points 51 minutes ago

Why are quantised LLMs harder to run than non quantised ones?

[–] DJKJuicy@sh.itjust.works 1 points 32 minutes ago

Well that sucks. I was really impressed as a novice to open weight LLMs with the ease of use for Ollama on Bazzite.

[–] brucethemoose@lemmy.world 4 points 1 hour ago

The localllama crowd has know this for years.

It happened faster than I expected, though; OpenAI/Anthropic hardly even got the chance to tighten the screws.

[–] FauxLiving@lemmy.world 18 points 3 hours ago (1 children)

The Western model was doomed to failure from the start. The only barrier to entry was being able to download thousands of TBs of internet archives/books and to have a lot of compute.

The math for these models isn't proprietary and most CS students are exposed to machine learning and neural networks while in school.

The only advantage western companies had was the ability to buy up the entire hardware market, pricing out domestic competition, and to use their politicians to manipulate trade policy in order to restrict sales of critical hardware to China.

Every US tech company has dumped billions investing in an unsustainable business model with the hope of buying a global monopoly by strangling competition.

China can destroy all of that by making their models open weight. The real money is in finding and implementing custom AI solutions... not in charging for access to the models. By having freely available models, they're making the barrier of entry as low as possible.

Not to mention that the insane amount of money being poured into hardware by US tech companies has created an environment where building fabs has a much shorter ROI, which also helps China's development in that sector.

US companies are playing Monopoly while China is playing Civilization.

[–] Cheebus@lemmy.world 2 points 1 hour ago

I still think this type of AI can lead to a worse state of living for most of us, but it’s better than the bullshit in the US.

Eat my ass Musk, Zuck, Altman and all you other fuck face tech bros

[–] nkat2112@sh.itjust.works 45 points 5 hours ago (2 children)

The article wastes no time getting to the underlying point in the very first paragraph:

Top executives at leading Western AI companies are increasingly warning about the safety and national security risks posed by Chinese open-weight frontier models. What they tend not to mention is that these models are improving rapidly and, because they are freely available, pose a serious threat to Western labs’ business models.

I found the following two paragraphs interesting:

By mid-2026, however, open-weight frontier models from Chinese labs such as Alibaba, DeepSeek, and Moonshot AI had nearly matched the leading Western models in intelligence and performance. Many companies have already begun building their AI systems on top of these free models, avoiding the high cost of closed-model APIs.

Because businesses can host open-weight models in their own private clouds, they can also avoid sending proprietary data to systems controlled by outside providers. Developers can fine-tune the models for specific needs, build applications and tools on top of them, and optimize them for their preferred infrastructure.

[–] Mosfar@sh.itjust.works 33 points 5 hours ago

Back to local computing is the way to go

[–] Buffalox@lemmy.world 28 points 4 hours ago* (last edited 4 hours ago) (2 children)

they can also avoid sending proprietary data to systems controlled by outside providers.

Not only proprietary but also personal or other kinds of sensitive data.
If you are doing health research on databases of personal health data, you should be able to guarantee the safety of that data.
That means you can't use the current American systems, because they've been shown to be insecure.
This would be a major issue in EU, where such data is legally protected.

[–] plyth@feddit.org 6 points 4 hours ago (1 children)

This would be a major issue in EU, where such data is legally protected.

Nothing a EU–US Data Privacy Framework can't handle.

[–] Buffalox@lemmy.world 7 points 4 hours ago

I absolutely agree that that agreement is complete and utter bullshit.
Hopefully the shift there has been to achieve IT independence from USA will mean EU doesn't give in so easy next time.

[–] cavitationfetishist2@quokk.au 3 points 4 hours ago (1 children)

These people are too rich to be punished by laws. They're the people laws protect not the ones they bind.

[–] Buffalox@lemmy.world 1 points 4 hours ago (2 children)

Not in EU, Eu has given fines to those big tech companies before and can do it again.

[–] cavitationfetishist2@quokk.au 2 points 1 hour ago* (last edited 1 hour ago)

More than their profits? Their profits over course of the shittiness?

Being less bald face and trying to hide your corruption better than America doesn't make you the good guy.

[–] grue@lemmy.world 2 points 2 hours ago

Fines are just the cost of doing business.

Wake me up when corporate charters are dissolved and executives are put in prison.

[–] Greg@lemmy.ca 13 points 4 hours ago* (last edited 4 hours ago) (3 children)

I just wish I could buy enough memory to run one of these models locally. Specially Kimi K3

[–] a1studmuffin@aussie.zone 2 points 1 hour ago (1 children)

The other problem is bandwidth. Even if you can run Kimi K3 locally, AI usage patterns tend to be very bursty. Having the scalability of cloud compute to run multiple agents and sessions at once makes it much more usable.

[–] Greg@lemmy.ca 2 points 1 hour ago (1 children)

My current workflow is passing a human written spec to an agent to implement with strict coding guidelines, architectural decisions, etc. The agent isn't making any decisions about the abstractions to use, it's just creating the objects and test suites. So I don't mind the slower bandwidth because I'm running the heavy agentic lifting over night with no need for human supervision.

But I fully appreciate that my workflow isn't the norm. In fact my workflow it's the exact opposite the AI grifters like Sam Altman are selling because it still involves a human with knowledge of the systems making different decisions.

[–] brucethemoose@lemmy.world 1 points 1 hour ago* (last edited 1 hour ago)

If you're running stuff overnight on a homelab, you can get away with using very large models as agents in CPU RAM.

Another option is a hybrid approach: farm "dumb" privacy-insensitive calls to an API you can subscribe to for like $40 a year, and let your local LLM do the orchestrating. This is what I do: I run MiMo locally, but also farm calls out to the exact same LLM in the cloud when appropriate... And I don't mind one bit. I'm happy to tip them for open sourcing it in the first place.

[–] brucethemoose@lemmy.world 1 points 1 hour ago* (last edited 1 hour ago)

How much RAM do you have?

I can run MiMo 2.5 at about 9 tokens/sec, on 128GB RAM, a 7800 and a 3090 in an SFF rig. That's a fantastic 310B model. I'm requantizing it right now, to see if I can speed it up with Dflash.

Still fantastic models can be run on 64GB or 32GB CPU RAM, as long as you have some GPU. We're awash in sparse models these days.

[–] Dionysus@leminal.space 10 points 4 hours ago (1 children)

Same, getting ~3 trillion parameters in consumer hardware is rough.

If Nvidia has any foresight they'll see the writing on the wall and start getting higher memory Spark style SMB inference machines, few people in the long run are going to pay retail API token costs,

[–] Greg@lemmy.ca 1 points 1 hour ago

Especially giving the lack of trust organizations rightfully have in big AI companies

[–] mctoasterson@reddthat.com 2 points 2 hours ago

If all AI really is classified as a "weapon" for export-control purposes, well... in the US possession of weapons is a protected individual right. The US gov should be training open weight models and encouraging self-hosting.

[–] yesman@lemmy.world 8 points 4 hours ago (2 children)

is their a meaningful difference between open weight and open source?

[–] melfie@lemmy.zip 1 points 18 minutes ago

Open weight is analogous to a compiled binary. Similar to how Windows is closed source, but still runs in your own hardware, whereas Linux is truly open source.

[–] herrvogel@lemmy.world 15 points 4 hours ago

Pretty big difference. An open weight model is a model that you can run on your own machine. You just download and it's yours to host and use. You don't need to have anyone host it on their own backend for you, the entire model is available to you to do that on your own. What you don't have is any control over or access to anything related to how the model was trained. You don't know what kind of data they used to train it, and how exactly they used that dataset. If you did, that'd be an open source model.

[–] cavitationfetishist2@quokk.au -4 points 4 hours ago (1 children)

Okay but what about the fact all of this is useless bullshit that will only make the world worse?

[–] keimevo@lemmy.world 8 points 3 hours ago (2 children)

It's far from useless. Overblown, for sure.

They made an awesome hammer that can solve many hammer-related tasks, but they're selling it like it can also cook, drive and keep the house clean.

Blame the companies, not the math.

[–] DJKJuicy@sh.itjust.works 4 points 2 hours ago

100%. These LLMs are fantastic tools for when you could use an LLM to solve a problem or make something better.

Using the right tool for the right job is really not a complicated concept.

The problem, as always, is marketing and hype noise caused by the people who see dollar signs.

[–] cavitationfetishist2@quokk.au 1 points 1 hour ago* (last edited 1 hour ago)

Every time I hear about these hammer related tasks I find out they kind of suck or drained lake Tahoe to translate a book that already has a hundred translations.

Or the hammer related task is that one scene from 'oldboy'.