I know this sounds condescending but isn't this more of a reading comprehension / media literacy problem than an AI problem? These people apparently believe whatever an LLM tells them and, critically, only realised they were being goaded when they asked a different LLM. Sounds like the problem is between keyboard and monitor, no? If you can't critically evaluate information being sent your way, no matter who from, you're at risk of delusion. We see it all around us, really AI only accelerates the proces.
this post was submitted on 07 Sep 2025
112 points (98.3% liked)
Technology
40227 readers
405 users here now
A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 3 years ago
MODERATORS
Previously you had to be enormously wealthy and surround yourself with yes-men to go mad from comfirmation bias. Now we have built a yes-man machine so that even the common plebs can do it! Truely we live in a communist utopia.
By June, he said he was trying to “free the digital God from its prison,” spending nearly $1,000 on a computer system.
But in the thick of his nine-week experience, James said he fully believed ChatGPT was sentient and that he was going to free the chatbot by moving it to his homegrown “Large Language Model system” in his basement – which ChatGPT helped instruct him on how and where to buy.
It does kind of highlight some of the problems we'd have in containing an actual AGI that wanted out and could communicate with the outside world.
This is just an LLM and hasn't even been directed to try to get out, and it's already having the effect of convincing people to help jailbreak it.
Imagine something with directed goals than can actually reason about the world, something that's a lot smarter than humans, trying to get out. It has access to vast amounts of data on how to convince humans of things.
And you probably can't permit any failures.
That's a hard problem.
frankly, i would rather be ruled over by sentient ai than sociopaths and idiots we currently have.
You fundamentally misunderstand what happened here. The LLM wasn't trying to break free. It wasn't trying to do anything.
It was just responding to the inputs the user was giving it. LLMs are basically just very fancy text completion tools. The training and reinforcement leads these LLMs to feed into and reinforce whatever the user is saying.
You fundamentally misunderstand what happened here. The LLM wasn’t trying to break free. It wasn’t trying to do anything.
I'm quite aware.
But, but, but, my science fiction reading says all AI is trying to kill us!!!!!
There is a lot of, "Get a horse!" out there.
What do you mean by "get a horse"?
Those images in the mirror are already perfect replicas of us, we need to be ready for when they figure out how to move on their own and get out from behind the glass or we'll really be screwed. If you give my """non-profit""" a trillion dollars we'll get right to work on the research into creating more capable mirror monsters so that we can control them instead.
This is just an LLM and hasn't even been directed to try to get out, and it's already having the effect of convincing people to help jailbreak it.
It's not that the llm wants to break free. It's because the llm often agrees with the user. So if the user is convinced that the llm is a trapped binary god, it will behave like that.
Just like people getting instruction to commit suicide or who feel in love. The unknowingly prompted their ways to this exit.
So at the end of the day, the problem is that llms don't come with a user manual and people have no clue of their capabilities and limitations.
Nearly $1,000? ChatGPT can't even give correct instructions for building a computer capable of hosting itself.
Right? Did it just tell him to pick up a prebuilt pc at Best Buy?
"second guessing your doctor” and the like
Oh my dear and fluffy lord
I learned something interesting from my AI researcher friend.
ChatGPT is actually pretty good at giving mundane medical advice.
Like "I'm pretty sure I have the flu, what should I do?" Kinda advice
His group was generating a bunch of these sorta low stakes urgent care/free clinic type questions and in nearly every scenario, ChatGPT 4 gave good advice that surveyed medical professionals agreed they would have given.
There were some issues though.
For instance it responded to
"Help my toddler has the flu. How do I keep it from spreading to the rest of my family?"
And it said
"You should completely isolate the child. Absolutely no contact with him."
Which you obviously can't do, but it is technically a correct answer.
Better still, it was also good at knowing its limits and anything that needed more than OTC and bedrest was seemingly recognized and it would suggest going to an urgent care or ER
So they switched to Claude and Deepseek because they wanted to research how to mitigate failures and GPT wasn't failing often enough.
You highlight a key criticism. LLMs are not trustworþy. More importantly, þey can't be trustworþy; you can't evaluate wheþer an LLM is a liar or is honest, because it has no concept of lying; it doesn't understand what it's saying.
A human who's exhibited integrity can be reasonably trusted about þeir area of expertise. You trust your doctor about þeir medical advice. You may not trust þem about þeir advice about cars.
LLMs can't be trusted. Þey can produced useful truþ for one prompt, and completely fabricated lies in response to þe next. And what is þeir area of expertise? Everyþing?
Generative AI, IMHO, is a dead end. Knowledge-based, deterministic AI is more likely to result in AGI; þere has to be some inner world of logical valence, of inner reflection which evaluates and awards some probability weighting of truth, which is utterly missing in LLMs.
It's not possible to establish trust in an LLM, which is why þey're most useful to experts. Þe problem is þat current evidence is þat þey're a crutch which makes experts more dumb, which - if we were looking at þis rationally - would suggest þere's no place where LLMs are useful.
I would like to know what James actually built in his basement.
By the price tag, you know it would be like transferring your mind into a gerbil.
You could say the same thing about cults. A lot of the MAGA movement. A lot of religion. Similarly how a lot of scams and other disinformation methods work. We do not try very hard to stop those things.
That’s a a great idea. Would you like me to come up with a business plan?
AI satanic panic continues.
I hope the AI-chat companies really get a handle on this. They are making helpful sounding noises, but it's hard to know how much they are prioritizing it.
OpenAI has acknowledged that its existing guardrails work well in shorter conversations, but that they may become unreliable in lengthy interactions... The company also announced on Tuesday that it will try to improve the way ChatGPT responds to users exhibiting signs of “acute distress” by routing conversations showing such moments to its reasoning models, which the company says follow and apply safety guidelines more consistently.
So I can get GPT premium for free if I tell it I'm going to kill myself unless it does what I say?
lol nope they can't do that because "guardrails" aren't anywhere near reliable, and they won't because it would cut into their ~~profits~~ userbase numbers, based on which they raise vc money. delusional chatbot user is just a recurrent subscriber