Selfhosted

Yes, were all on c/LocalLLaMa

I'm running qwen3.8-next-flash on a rx7900xtx plus 128gb of ram. It serves my hermes agent. I direct it through matrix. I also forward an endpoint to my VPS to use for anything. Mostly opencode.

I recently gave Strata a shot for my local AI. I'm very impressed with the performance they have eeked out of my hardware. I've got a rather beefy gaming desktop I built as an AI server back when the current craze started, before prices started going up, but models up until this point really failed to take full advantage of that hardware. Strata (which uses a Qwen model) both simplified the setup process, and it maxes out the hardware when running.

Strata acts like an OpenAI compatible API server, I then have Hermes as a "harness" to make an "agent", which I interact with on "buzz", which is like discord, but made for bots. Although Hermes also supports discord and several other chat platforms.

If you have the hardware to run Strata, I recommend setting it up, it comes with a basic chat interface, which you can then grill to assist you with setting up Hermes and whatever else.

One thing I didn't mention, is security. I highly recommend you treat AI as unpredictable, unreliable, and capable of being taken over by bad actors. This isn't a problem for a pure chat interface, but Hermes gives the AI tools to do things, and those tools could be used to take over your computer, scan your network, and create all kinds of havoc. For my part, Hermes is running in a virtual machine which is on an isolated network. I keep tight control on what it does and doesn't have access to in my network main network. That may not be something you have the hardware or knowledge to do, at the very least I recommend running Hermes in a docker container, and not giving it access to your personal desktop nor credentials to things like your email account.

I wish I could. I don't have the hardware for it.

We did the maths. Didn't seem worthwhile and the payback period was a very long time

If you haven't seen it already, take a look at !localllama@sh.itjust.works

Yes i do.

I esperiment a lot, but my stable setup is Qwen 3.8 27B Q6_K on my home server. Hardware is Ryzen 9 5950x with 64gb ddr4 and two old but effective NVIDIA RTX A4000 for a total of 32gb VRAM.

Qwen 3.8 is the best model I can run in terms of output quality. It can perform all tasks pretty well and it's reasonably fast at 500ts prefill and 30/40ts processing. There are much faster models but honestly not as good.

I can also run gemma4 all on CPU, with no GPU at all.

The key point is having both GPU on pci-e CPU, not on chip set bridge, so I am using a PCI riser 8x8x to share the 16x slot between the two GPUs. Inference speed is unaffected, but prefill benefits immensely.

Software side, use llama.cpp on Linux, you get the best performance possible at the expense of a bit more complexity compared to other engines.

From a power consumption perspective, I tined down my CPU governor and maximum frequency, and also the GPUs maximum watts. At ide the system is around 80w. At peak, it's about 280w with the limiters, and inference speed doesn't suffer too much. Without limiters I can easily reach 650w.

Edit: I run llama-swap in front of llama-cpp, and openwebui as agent manager/ web GUI. I also have ComfyUI installed because why not but seldomly use it.

You can read more of my setup on my wiki https://wiki.gardiol.org/8-llm-and-ai (no ads, no monetization whatsoever because fuck a hobby should not be a side gig)

What does it do? I thought about trying this but I literally never have a need to generate text.

This is something I will do probably within the year, but it feels to me like the area is still moving quite rapidly. The model will naturally depend a bit on the hardware. I'm looking at used AI cards because it looks like currently for about $2k you can get one that will run ~350B models, and GLM is nice in that space, though I will experiment with other models once I can run them.

Things I will do:

• try to replace Alexa for voice activated smart home management — this will let me go locally controlled I hope (my wife is big on convenience and could give a fuck about privacy so the solution has to be solid).
• a personal journal so I can have it remember things so later on I can ask things like when is the last time a friend and I hung out, or what was that orchard we went to last October, or what was that great B&B we stayed at, let's do that again. This is likely the thing that would require a good model — everything else is pretty simple orchestration, aggregation, and summary. Arguably it's the thing that would make the biggest difference in my life, too. There is a lot of beauty in life but my brain is wired to constantly churn solving problems and be irritated and I rarely stop to appreciate the moments or remember the accomplishments.

I have a ton of ideas for conveniences AI could do if it had total privacy and security. In short, orchestrate all of the tools and knowledge that I already rely on constantly. Integrate calendar and private info like SSNs, VINs, who has what doctors for what, or who was sick when, smart alerts like the overnight temp will be below freezing next week so make sure the house is winterized, or I wanted to see this movie and it's releasing or will be leaving theaters soon — oh and next weekend you're visiting the grandkid, so it's either now or I'll remind you again when it comes out on streaming.

Just having a personal knowledge base always on and totally private would be handy. It would be like having a personal assistant maybe concierge if it could be trusted to that degree.

• 96gb ddr5
• intel i7 12th gen
• RTX 3060

it also doubles as my media ripper. I initially built it specifically for that before AI was even a thing.

I run too many models to count really. everything through CLI because I'm dissatisfied with all the sloppy UIs.

32GB RAM, RTX 2080 8GB VRAM, I use Tiel Coder 35b-a3b (it's a fine tune of Qwen 3.6) with Zoo Code

I’d like to, as I currently drop $20/month on Claude without perhaps using it to the extent that my money would let me, but what’s “good” that can run on a 16GB RAM, i5 12th gen, 3070-equipped desktop?

Yes, with OpenClaw I can expose my agents via discord apps on my home discord server, I like that route because of the rich media and UI options (agents can give me drop down selector lists and approve/reject buttons). I have two 24gb Nvidia cards for local models usually Qwen, Mistral, and nemo based models for different purposes. I can chat with the agents anywhere I'm logged into discord and I expose privileged control surfaces and nodes via tailscale and family/shared endpoints via caddy and cloudflared

I run lemonade on my strix halo box, I change the models I use, but the qwens are pretty good.

I access it via Tailscale on my phone, then I have a page I can use. The Lemonade interface forces you into the app, so you’ll want some other interface for your LLMs, as the app is junk.

I used to run llama.cpp and llama swap, but lemonade combines them and makes managing different ai backends easier.

I have a Qwen model running on a Mac Mini with Ollama - it’s reasonable for Home Assistant, responding to iMessages and similar.

Three models my self, qwen 27b as a dev assistant, a small model rigged up to handle voice input as an assistant that's connected to my intercom system though out the house. Mostly to run my lights/music/audiobook play back. It's fun having jarvis control the house after all. Then a image generator.

Llama and swap on a 7900xtx.

I haven't gotten around to setting up phone access but wireguard for remote access form work in my laptop.

Honestly it's all basically over priced toys. I'm baffled that people rely on this stuff. I look at it the same way people do with model airplanes or RC cars.

Over priced toys that are hell of a lot of fun. Break frequently and require more care then they are probably worth to use safely lol

90€/mo to self-host something? Lol

Wonder if anyone has looked at power costs of running local ai. Some things sound nifty, but if it's costing me 20 a month it doesn't seem worth it

Sure. I have Home Assistant with STT and TTS. And I use KoboldCPP to experiment with smaller LLMs. Nothing fancy, it'll run on my old laptop, albeit very slow.

I find the idea interesting. However I wouldn't get the commercial service you are advertising. Maybe something that can run on an older CPU as well. Doesn't need to be fast, but if it uses up the idle compute I have on my rented server, then why not.

Hardware is a 5070 12GB VRAM, an AMD 5950x and 64G of DDR4 running on Nobara (Fedora spinoff).

I use 3 models for most things: Ternary Bonsai 27b and Gemma 4 E4B Uncensored Q8_0 for everyday stuff, and Qwen Agentworld 35B A3B Q8_0 with CPU offloading for tasks that require to keep the model on track for long periods of time.

Everyday performance is best with Bonsai for me - it comfortably fits in my VRAM and is noticeably better with staying on task than similarily sized models. But if i want it to work overnight on a task without issue, the lower speed of the agentic Qwen is outweighed by its capability to do this.

They are allowed tool calls for websearch and reading webpages and have a sandboxed directory for R/W operations, but anything regarding command line and R/W access in other areas has to be manually authorized by me (i'm not willing to play with fire here).

They are currently not reachable from outside, but i am currently building a smaller machine as a home server, which will then be using a 3070Ti and 32G RAM to provide probably the Bonsai Model for me on the go per VPN.

The smaller models are here for quickly creating documents i do not care to do myself, RAG for technical documentation, creating quick python scripts for automating small tasks and teaching me RegEx and Python(mostly by analyzing the LLM output and playing around with it) and also some RP as NPC in a fantasy setting. The agentic model is currently working on cataloguing my games collection (around 36000 titles) by looking up games, their genres and descriptions and filling in blanks in the database. (Funnily, this task was refused by any model freely available because of the size of the database. Even the local models moan and bitch that it takes so long. No shit, this is why i outsource this!)

I am running deepseek-r1:1.5b on a laughably old dell laptop (circa 2011) cpu only (no gpu) and 16 GB ram. It is slow but works as a general assistant for Newelle. Mostly proof of concept but so far it has helped me proof read documents and tweak its own setup. It al also runs clean up and organization jobs on my laptop to manage the mess of old files on it.