Why is the opening story so vague? Middle school counselor gets an automated alert, calls parent, calls police??? No explanation of what caused the alert? Is this supposed to be presented as a success? Then it says "She believes that the interaction built trust between [counselor] and the family."
And other monitoring systems "which would refer the student to in-school suspension " for... reporting "my mom beat me with a pole" or "my uncle touches me"?? I'm so confused by all of this. I don't use AI at all, and never touched AI therapy. I must be so far out-of-the-loop. None of this makes sense.
Removing AI LLM slop from schools is absolutely the first step, and would be impactful at that. These kids are being forced onto AI through their schools. If we win this battle, one day we'll look back at these "historical figure" chatbots the same way we think of candy cigarettes.
I want to iterate that school curriculum is incredibly controllable at the local level. Check your school board. Rally for a No AI policy in lessons, and perhaps in teaching materials. This at least is one step we can take.