this post was submitted on 07 Sep 2026
476 points (97.8% liked)
Technology
87897 readers
4242 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Ohhh lol. Yeah it's Llama-swap, running llama.cpp for now, but might add vLLM to the llama-swap config to experiment with NVFP4.
I mainly use MoE models so I can get decent speed while using a 150-200k context window. My go-to model has been Qwen3.6 35B-A3B for a while. I tried Qwen3.8 27B, but it was too slow.
Gemma4 26B-A4B also runs nice and fast, but I generally get better results from Qwen3.6. I don't remember exactly how much CPU offloading is happening, but it's not much. As long as I can get like 40-50 tokens/sec, I'm usually satisfied enough.
For the coding harness, I've been running Pi in an Apple Container (sort of like Podman, but better isolation in a microvm). Though, I recently configured VS Code to use LLMs on my server, and it was actually pretty decent. Still need to explore a bit more, but so far VS Code's AI capabilities seem much better than they were a year ago (they seemed way behind, back then).
Also, I don't connect any harness directly to llama-swap. I have another container running Caddy, which acts as a gateway to AI providers. For other services (e.g. OpenRouter), the API key is injected in the Caddy container. I don't like having API keys or secrets anywhere where LLMs can read them. It's not so bad for my own self-hosted LLMs, but not cool to send secrets to a server owned by someone else.
How has tool use been for you? I struggled a lot with tool use with Gemma and Qwen, to the point where I needed to build a healing layer.
Regarding the coding harness, I was looking for something CLI-based or JetBrains-based, and I haven't had much luck getting my local llama.cpp models playing ball with OpenCode. They keep losing context and misusing tools.
I'm not too familiar with Apple containers as I'm running a full Linux stack, but I'll give Pi a try, seems interesting! Does it work for coding tasks or is it strictly an "orchestrator"?