486
Teen killed himself after ‘months of encouragement from ChatGPT’, lawsuit claims
(www.theguardian.com)
this post was submitted on 27 Aug 2025
486 points (96.4% liked)
Technology
74873 readers
2789 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Not encouraging users to kill themselves is "ruining it"? Lmao
Thats not how llm safety guards work. Just like any guard it'll affect legitimate uses too as llms can't really reason and understand nuance.
That seems way more like an argument against LLMs in general, don't you think? If you cannot make it so it doesn't encourage you to suicide without ruining other uses, maybe it wasn't ready for general use?
You're absolutely right, but the counterpoint that always wins - "there's money to be made fuck you and fuck your humanity"
Can't argue there...
It's more an argument against using LLMs for things they're not intended for. LLMs aren't therapists, they're text generators. If you ask it about suicide, it makes a lot of sense for it to generate text relevant to suicide, just like a search engine should.
The real issue here is the parents either weren't noticing or not responding to the kid's pain. They should be the first line of defense, and enlist professional help for things they can't handle themselves.
I agree with the part about unintended use, yes an LLM is not and should never act as a therapist. However, concerning your example with search engines, they will catch the suicide keyword and put help sources before any search result. Google does it, DDG also. I believe ChatGPT will start with such resources also on the first mention, but as OpenAI themselves say, the safety features degrade with the length of the conversation.
About this specific case, I need to find out more, but other comments on this thread say that not only the kid was in therapy, suggesting that the parents were not passive about it, but also that ChatGPT actually encouraged the kid to hide what he was going through. Considering what I was able to hide from my parents when I was a teenager, without such a tool available, I can only imagine how much harder it would be to notice the depth of what this kid was going through.
In the end I strongly believe that the company should put much stronger safety features, and if they are unable to do so correctly, then my belief is that the product should just not be available to the public. People will misuse tools, especially a tool touted as AI when it is actually a glorified autocomplete.
(Yes, I know that AI is a much larger term that also encompasses LLMs, but the actual limitations of LLMs are not well enough known by the public, and not communicated enough by the companies to the end users)
I hope that's true, the article doesn't mention anything about that. I'm just concerned that he was able to send up to 650 messages/day. Those are long sessions, and indicative that he likely didn't have a lot going on.
I definitely agree that the public needs to be more informed about LLMs, I'm just pushing back against the apparent knee-jerk assignment of blame onto LLMs. It did provide suicide support info as it should, and I don't think providing it more frequently would've helped here. The real issue is the kid attributed more meaning to it than it deserved, which is unfortunately common. That should be something the parents and therapist cover, especially in cases like this where the kid is desperate for help.
Very fair. Thank you!
I'm not gonna fall for your goal post move sorry
I'm honestly at a loss here, I didn't intend to argue in bad faith, so I don't see how I moved any goal post
As far as I know, magic doesn't exist, so words are incapable of action & can't actually kill anyone. A person who commits suicide chooses it & takes action to perform it. They are responsible for their suicide even if another person tells them & hands them a weapon.
These are merely words on a screen lacking force to compel. There's no intent or likelihood to incite imminent, lawless action. Readers have agency & plenty of time to think words through & reject ideas.
It's hardly any different than an oblivious peer saying the same thing. Their words shouldn't create any legal obligation, and neither should these.