disclaimer: This isn't an issue specific to the company involved, more than likely they were fooled into believing AI assisted customer service would work, just like everyone else.
A centaur is the idea of a human directing the AI horse. A reverse centaur is the AI being carried by the human, telling them where to go.
tl;dr It doesn't work and even if it could, everyone is definitely using it wrong. AI has created an even worse bad solution to a difficult problem. LLM's and chatbots are replacing critical thought and empowered employees (if there were any) with Douglas Adam idiots boxes directing people unable to point out the machine isn't right. Yeah.....
I wanted to change my email address with a prepared meal service called CookUnity. Going to settings I didn't find any option to do so.
After emailing support, "To change your email address you need to cancel your account and create a new one."
Replying back "Are you absolutely sure? Also does anyone over there realize how unhinged (this is)?"
AI responds back in the affirmative that I need to cancel my account and create a new one.
One last question "Before I do this, I just want to be sure I won't be punished for doing this."
This time a human proceeded to phone bomb me with repeated call after call until I could answer. Their accent and command of the English language suggested they were some unfortunate soul in a call center somewhere in god knows where. My brief conversation gave me some idea of how the "Customer Experience" is implemented.
Unfortunately it turns out later, with no surprise, I was indeed punished for creating a new account.
To be clear, this is not some Greek tragedy solely inflicted upon me but is definitely happening everywhere with everything and it has been like this for sometime, but now these AI companies have added themselves (and their service fees) to an already broken system.
I speculate the process is like: Alice has a problem, she contacts customer support. An LLM processes her message, generates a response as per the policy (or lack thereof) in the system prompt, and then Maria the reverse-centaur located who knows where on Earth does a quick QA and maybe some minor adjustments. Maria's income is very likely weighted by some inane metric like closed tickets which actually doesn't work for promoting quality and instead just pushes quantity.
The response Alice gets back may or may not be helpful. If the issue was Alice not understanding how to use or not knowing how to achieve something with the service, maybe the LLM was able to rephrase things so it makes sense. Alternatively if the problem is a policy conflict or an edge case problem, the LLM is going to give a beautifully written, magnificently marked up, ultimately useless response. In this case Maria may not be able to escalate the issue because perhaps that is another inane metric? How many tickets did she escalate? This feels a little insidious as this unfortunate reverse centaur can't actually do anything but get in trouble with their metrics or click Approve.
Beyond my lament that AI sucks, it is worth noting that all of these companies are probably infected with these AI services so deeply that when the prices go up, the companies won't be able to escape because it will cost real money now for the possibility of saving money later. That's way too risky!