this post was submitted on 17 Jul 2026
489 points (99.2% liked)
Fuck AI
7560 readers
1150 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Wow, this is lazy. I feel like you could probably set up an AI agent thing to check every case it references to make sure it is both real and relevant. But they were so lazy not only did they not check it themselves, they didn't even have a clanker check it.
I don't think LLMs are great at figuring out if something is written by an LLM, or factual.
I wouldn't have it do that. I'm thinking more along the lines of an AI that drafts legal documents.
I'm not like an agent engineer or anything like that, and I'm sure there's more efficient ways of having an LLM draft legal documents. But so long as you can get it to reference primary sources there should be a lot less cleanup work to do.
You need to format a database query from the file. Then you need to compare what was said in the file to the actual text. Both are things an AI is good at.
You dont need to - you just have to make sure you set up one or two to find holes. Make it 50 points per hole, and 1000 bonus if all is proved wrong. Make two compete and let them know
And then you can have more that check those results for faults!! With only 500 agents working full time you have quite a great setup
"Who watches the watchers?", that problem is far older than both me, you and every LLM.
Or, hear me out, we don't use generative AI for anything related to human welfare.
Or, hear me out, we don't use generative AI for anything ~~related to human welfare~~.
Asking a chatbot to stop lying is about as useful as asking a river to stop flowing. It won't happen unless you do it yourself.
I lay out how I would do it in another comment. I think it would be pretty easy to get something rough draft quality pretty easy. The difficult problem is compliance, you would probably have to use either local or legal work approved LLMs. But I'm saying they actually just used a chat bot instead of an agent that could give you better results. It's pretty easy to get a clanker not to lie, use it as a tool to rearrange text and don't let it be creative.
The law is a hundreds of billions of dollar annual industry, just in the US. It was worth 367 billion just in 2022. An AI that flawlessly checks over a legal brief would probally be worth billions/year because that means it could also write them flawlessly. Since its so easy, you should probally whip it up and make your billions.
Hell, you seem to have a way to solve chatbots persistent error rate that trillion dollar companies can't fix, so you might actually have a trillion dollar answer. Why stop at just law if "its so easy to make a LLM not lie?"
It's a saturated business, there's dozens of companies already doing it. I don't like AI, I think it's destroying a lot of parts of our life. But it's here and it's only going to sink it's claws deeper. Also I'm not describing chat bots.
And I'm saying that lying is inherent to the technology. Generative AI uses probability to generate output. It's constantly rolling dice when determining output. Any data fed into it is used to calculate odds on what words should follow other words.
It's like making a smoothie and then trying to consume specific vitamins. No matter how good your smoothie making machine is, that's gonna be an impossible task because blending everything together is inherent to making a smoothie. You'd be much better served just eating individual bananas to get your potassium intake.
And I agree with you. When you let an AI be creative it will lie. But if they are reading documents they generally don't. That information is now a part of their context and it will reuse that information. It's a good fuzzy search. It is however terrible at a lot of other things.
Edit: it's like a plagerism machine! Which in this one case is probably good
If you feed a chatbot the sentences, "Bananas are a good source of potassium. Bananas are not a good source of protein." There is a good chance it will spit out the phrase, "Bananas are a good source of protein." because the phrase "good source of protein" is in the original data it was fed. There will always be a significant chance of spitting out this phrase.
No matter how much you improve the AI, because it uses probabilistic generation, it will always be cobbling together sentences without any regard for what the overall sentence means. An intern will eventually learn case law, an AI agent will never be trustworthy. When you ask an AI agent a question, it will always lie to you about being certain of its answer because its answer is built upon a roll of the dice.
For "real", you don't even need an AI. For "relevant", I wouldn't trust an AI. I don't think AI can solve that problem.