Selfhosted

Yes, I got a Strix Halo machine before the RAM price hike and use it to run all my ML stuff on it.

Currently using llama-swap with llama.cpp/ComfyUI and opencode/Open WebUI as frontend.

I'm running Qwen3.6-27b, Voxtral Mini 4b, Piper and Qwen Image. Also, some embedding and reranking models.

I use them for:

• Tagging and classification of my documents in Paperless
• Home Assistant (voice assistant)
• Translations (both text and image)
• Transcriptions
• Some light coding and debugging
• Avatar/Backdrop generation for DnD sessions

i don't use it at all, i do want some selfhosted speech to text model (whisper?) but my computer is ancient so it would be awfully slow. i have some multi hour audio recordings from presentations, would be nice to have them in text and searchable..

No, too expensive. I wish I could but it doesn't make sense financially for me right now, it is much cheaper to buy openrouter credits from time to time

I've been running ministral on CPU on a home-server: works pretty nicely, not very performant for everyday tasks and the savings were not sufficient for it to make sense. It still was cheaper and faster to just use Mistral API and get better models.

Yeah, mostly for translation purposes.

I think I currently have gemma 4 set up.

I'm running dwarfstar which is a 2 bit deepseek v4 flash. It's quite capable even at 2 bit.

I have the setup, never found a use for it though.

I tried but I only have 16g of ram and it wouldn't complete a thought alas

I tried Qwen 3.6 a3b and Gemma 4 a4b, but both were too stupid for everyday work.

I ran through lmstudio because it really eazy, I ran some kind of qwen 3.6 27b imatrix neo code DI, it is the best local model for coding I tried, I think it can be better than some cloud model

Why would I?

An aside for anyone reading this:

https://sleepingrobots.com/dreams/stop-using-ollama/

And that barely scratches the surface. Please.

Use anything but Ollama. Even APIs.

Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.

I set up ollama on our thinkstation in the lab and I use it for looking up documentation, generating readmes, searching papers, and sometimes coding when I know what to do but don't feel it is worth it to spend time on it myself. So basically the chat with web search.

I run Handy with Parakeet for speech to text, and home assistant with Whiper for the same. Whisper+ on my phone.

I think that counts. But I have more relevant and useful things to do on my hardware and no 2000€+ to get LLM-capable hardware 😂

Yeah, I'm using qwen 31b a3b on an amd 9070xt requires a bit of cpu offloading, but still plenty fast. Using it wall llama.cpp. Combine that with some mcp's such as ddg-search to make it truly useful by actually being able to search online.

I mostly use it for small tedious tasks with well defined inputs and outputs. For example when hyprland recently changed from their own configuration language to lua. At first I started going line by line translating my config to the new lua language until I realized oh wait this is exactly the type of thing that ML is useful for. Going from the well defined hyprland configuration language to their also well defined lua syntax. It banged it out in less than a minute with only a single mistake which I easily fixed. The mistake it made was that it forgot to translate the comments to lua. It did it in less than a minute and worked first try. Where as I had made several typos and gotten a few lines wrong when I was doing it by hand.

Not to say that I couldn't do it. I would have gotten it done in about half an hour, but less than a minute is a lot faster.

I also used it to transform a bunch of unstructured data into json data, so that I could then use purpose built tools like jq to parse that. If I'm having trouble finding certain information. I'll ask it to find me some resources to look at.

Basically small well defined tasks and parsing data is what I use it for and it seems to be pretty good at that.

What I don't like is the way companies try to market it to people. I don't believe people should be trying to summarize emails or messages from loved ones, writing essays or any other creative tasks for the most part. Translating is okay. I don't expect a machine to be able to decide things for me or to be some filter between me and others.

Yes. Openwebui/ollama for LLM, comfyui for stable diffusion. I just dick around with it as a toy.

Found vLLM to be the most efficient local runtime service. And "ray" as a good (but complicated) way to distribute the load: https://docs.ray.io/

I recently gave it a try with qwen3.5 and deepseek coder v2. I have a RTX3090 and these are the largest models that can run comfortably on it.

Conclusion, they are both fucking useless. Free tier claude runs circles.

Nope.

I've thought about it, but I actually could never think of anything I would do with it.

I started out playing around with code generation using Ollama/open-webui and qwen 2.5 coder 14b on a 3060 12GB, but ended up on a winding journey with an ex datacenter card called the AMD V620. Its roughly equivalent to an RX 6800XT, but with double the VRAM. At this point i've really done nothing productive with it but learned a lot about bios settings, GPU/ROCm drivers, and custom fan solutions/PWM controls trying to get it setup and optimized haha.

It's pretty sick though, that amount of VRAM with 512GB/s bandwidth can run Qwen 3.6 27B dense with 100k context window at 20 tokens/sec in LM studio. Draws 300 watts at the wall on my ITX chassis (idling about 30w).

I've been dabbling in building an aviation weather and field condition report application using this, but my next step is to rebuild my VS Code environment into a new machine. I'm kinda enjoying just fucking around with building the hardware too though