Selfhosted

Well, I don't exactly host AI. But some of my software uses AI and/or machine learning. My photo gallery does face detection, I've installed text to speech and speech to text. My Home Assistant has a voice satellite (which is a poor-man's Alexa because I lack the hardware to do voice recognition in realtime). And I also regularly try some large language models and chatbots. But I don't have any real application (yet). And it's slow without a proper GPU. So I'm more or less just messing around. Currently that's with Ministral 3.

Yep.

I have a RTX 3090 + 128GB CPU RAM.

Currently I run my own custom IQ3_KT quantization of MiMo 2.5 300B, and it’s crazy good. It’s better than API models from not that long ago, and it’s served at about reading speed.

Never thought I’d ever run such a thing on my lowly desktop.

For quick scripts or code assistant, sometimes I use Qwen 27B (another custom quant, currently experimenting with exllama). Or Gemini 12B for messing with image/audio input. But TBH MiMo 2.5 with thinking disabled is smarter than 27B with it.

…And honestly, I use GLM 5.2 API a good bit.

I was lucky enough to get a yearly subscription for like $30, 6 months ago. I do self host the UIs or whatever takes the prompts, though.

Myself - I've self hosted LLMs before, but with only 4-8GB vram (depending which card is in place), I can't run the good stuff at acceptable enough speeds.

(Don't @ me - I know all the tricks with turbo quants, spec decoding, MoE etc. 192GB/s is 192GB/s)

I do use Handy (STT) which is amazing (my fingers are arthritic and typing hurts after a while).

My personal use case for LLM is quite simple - a trumped up super google and / or self reflection / journalling / sound board. Despite being glib about it, that's actually very useful to me.

Work wise, I use the big winking orange asshole (Claude) when I have to. I have moral tension with with it, so am seriously looking at other options. I hear good things about GLM 5.2, but if I can't run Qwen 35B at any kind of decent speed, well....self hosted GLM is a pipe dream.

I have a simple slow model running on CPU in my cluster for karakeep. I've tried running a variety of models on my 7900XT but even with 16GB their performance just isn't there. My new work m5 Mac book with 48GB of ram is the first time I've seen usable performance for local models and it has been pretty impressive.

I've tried just about most of the small models. Tried NanoClaw. I just don't have the equipment necessary to pull that off and make it a worthwile, in house tool rather than an in house oddity. I really, really want to tho. So much so that I have been looking at what it would take to accomplish that, which seems to be at the $4k to $5k USD range. The sweet spot for GPUs seems to be at the 32 gb level. It is pricey, but hell, at my age, I figure wtf....I should treat myself. Whats wrong with that? If I do pull the trigger, I want it to be a LTS type computer like I built 15 years ago and is still running like a champ today tho it's probably worth less than a quarter of what I had invested. So, I'd probably overstock it to the max.

I've fiddled around with a few models on ollama and opencode but more for the sake of seeing what I can run as ive yet to really find a use for it in my home usage.

I’ve got ollama setup with whisper and piper and a HA voice PE, but I honestly haven’t gotten around to configuring much yet. Most notable thing was being able to use the wake word to start a timer, but it was pickier than old Siri about the precise wording.

The short answer is no.

I have played around with ollama and whisper. It's just too slow to be practical. The cost of the hardware is preclusive.

That said, I do selfhost openwebui and use inference end points from huggingface and ovh.

I've never used chatgpt or claude and I have to wonder whether those alternatives are really as terrible as the models available on huggingface. The output is always super plausible but usually just plain wrong.

I use my gaming rig to serve up qwen3.6-coder to Open Web UI and that's been very successful in helping me refactor my home lab to be more effecient and easier to support. Over the years of building my server I got everything working, but lets just say it's a bot of a mess and a lot of shortcuts were taken.

I plan to look into ComfyUI soon but I do that have much of a use case for it at the moment.

I installed LM Studio just for fun on a 6800XT. But it was even less useful than the web-based ones.

I've played with it for Home Assistant integration, but I just dont have much interest in it, the whole thing is too inefficient at the moment, and the tiny models that can run in a few gigs of system ram on an ipgu or npu arent good enough in quality or speed to rely on.

Hopefully some future generation micro-models will be more useful for the way I want to use it (aka , ultra light, no dedicated hardware etc.), but for now it's a lot of compute resources, plus heat and energy for a gimmick.

My server is way to weak for that unfortunately. I run some llms on my laptop with ollama but it's not particularly effective. I use it to run dolphin series models when k need an uncensored LLM I have tried running some of the coding models but they just aren't smart enough on my level of compute for any useful work so Ive ended up just paying api prices on open routers.