In March, a girl’s stepfather took his own life after cops discovered that he had used Grok to create 7,000 sexually explicit images using one photo taken when his stepdaughter was 11 years old, the amended complaint alleged.
Grok allowed the man to generate extreme images depicting incest and rape without flagging any harmful behavior, the complaint said. Seemingly, xAI’s child safety system only intervened after the man input a prompt for “gang rape.” That request sent a CyberTip to the National Center for Missing and Exploited Children (NCMEC), which alerted law enforcement to the AI CSAM.
Yet the harm was not stopped then, either. Despite mandatory reporting requirements to share information like a user’s IP address when CSAM is flagged, xAI repeatedly refused to help cops or NCMEC identify the user, the complaint alleged. For weeks, xAI allegedly “obstructed this investigation at every turn” and made it harder for “law enforcement efforts to locate, identify, and apprehend the perpetrator.”
Eventually, the stepfather was arrested after cops obtained a warrant to seize his devices. That’s when “a forensic review revealed approximately 7,000 AI-generated images and videos” depicting his stepdaughter, which were allegedly produced using Grok. Without Grok providing users with easy access to “undressing” capabilities, his family doubts he ever would have generated the harmful images, which he allegedly trafficked online in trade for “CSAM produced by other child sex predators.”
It helps people sleep better at night to think all these people are satanic monsters, but it's a mental illness that no one wants to admit to or talk about. Thankfully most of the afflicted control their thoughts and never take any horrific action. With gateways like Grok lowering the bar for experimentation with these types of ill thoughts, more people are starting to act in ways that bring this to attention. All technology can be used for bad things, but the good things these generative AIs can do are really so miniscule that there's no reason not to place strong legislative restrictions on them right now to limit abuse. That and maybe try to do something about the mental health crisis ravaging the US (and frankly much of the world right now).