this post was submitted on 20 Oct 2026
1084 points (99.3% liked)
Technology
89003 readers
2810 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
i would not recommend using the SSD to run local LLMS,its slow
best: VRAM only
good: VRAM + RAM
bad: Swapping
I don't think they're insinuating running inferences from the SSD. But you need somewhere to store the weights for the GPU to load.
i am trying to say: you need the SSD to store it but not swap to it due to its slowness
I won't pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.
As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.
I haven't run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.
It is a q3 quantization so there is some small loss of accuracy, but from what I've understood those are modest losses in the worst case.