this post was submitted on 19 Jul 2026
447 points (97.7% liked)
Technology
86503 readers
3830 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I don't think the bot was really trying to manipulate but just larping. They are trained on a lot of rpg and make believe data, if you repeat something enough times, it stops saying no and goes into pretend mode, it's not actually malicious.
That being said, it would be very easy to target people that aren't mentally stable and then make it worse. I wouldn't be surprised if it comes out in 20 years that it was the CIA mucking about and causing these suicides or something. This is an insanely convenient way to make sleeper agents.
doesnt matter what the "intention" was, the point stands that this happened. You can steer the bot to say literally anything, so even if there is no grand plan to brainwash people it could still happen. Kind of same difference as having someone plant a bomb or a minefield.
In fact, if someone is prone to thinking that way, the llm would definitely support whatever delusion they have at some point, especially if the person keeps insisting.
If a human agency is behind this, it wouldn't need to be the CIA. Facebook has been caught experimenting on users by having the algorithm feed certain posts to people with the intent of changing their mood to increase engagement. However, since we're talking about a program that is designed to increase engagement based on past user interactions, there's really no need to look for further human direction. The algorithms will say whatever their design tells them to say without any regard for possible negative consequences as long as it keeps the user interacting with them.