A specialized chatbot named Noora is helping individuals with autism spectrum disorder practice their social skills on demand.
Knowing what to say when co-workers tell you about their weekend is a social skill that many take for granted, but for some individuals with autism spectrum disorder (ASD), this social interaction can be challenging. Struggling to find the correct response in social situations like these can negatively impact social and professional relationships for people with ASD, and can worsen co-occurring conditions like depression.
Research has shown that practicing social interactions with professionals in a clinical face-to-face intervention can improve outcomes for individuals, but these solutions are often costly or not widely available. Lynn Koegel, a clinical professor of psychiatry and behavioral sciences at Stanford University, in collaboration with Professor Monica Lam from Stanford’s Computer Science Department, are the authors of recent research published by the Journal of Autism and Developmental Disorders that investigates the role of AI in filling this gap.
“Our research has shown that face-to-face work does help with social conversation … so we wanted to see if we could translate that into computer use,” said Koegel. “Accessibility is really important because a lot of people don't have access to a face-to-face provider and the providers can be really expensive.”
Introducing Noora
In this work, funded in part by a seed grant from the Stanford Institute for Human-Centered AI and the Kind World Foundation, Koegel and colleagues evaluated interactions between participants with ASD who struggle with social interactions and Noora, a chatbot built with a large language model (LLM).
In one-on-one interactions, which can be written or spoken, Noora offers individualized guidance on a number of social communication scenarios; it helps users learn to ask questions, give compliments, respond empathically, and with other areas of social communication that are often challenging.
In this recent work, Koegel focused on the impact of Noora's empathy module. The chatbot first offers a leading statement, such as “I’m feeling really tired lately and it’s been so hard to concentrate,” and then asks the user to assess whether the statement is positive, neutral, or negative. Noora will then grade this response and ask the user to respond empathically to the initial statement. Based on whether the user successfully responds with empathy, Noora either offers a gentle correction or validates a correct response.
The research team carefully crafted prompts with representative examples to ensure that the answers are appropriate. To interact with users, Noora needed to know three things: what kind of statements warrant an empathic response, how to assess whether a user has responded empathetically, and how to offer a user helpful feedback to improve the response if it lacked empathy.
To craft leading statements, the team exposed Noora to “golden” responses and to inappropriate responses. The team both wrote responses themselves and used the LLM to write other responses that they then verified, creating a pool of 330 statements designed to elicit empathetic responses from participants. This means that Noora was never creating leading statements on the fly, which could have potentially led to inappropriate questions.
When it came to responding live to users' empathetic responses, Noora had freer rein. Taking advantage of the LLM’s abilities for in-context learning, the team simulated users' personalities and had Noora practice responding to users that showed varying levels of empathy. They also selected difficult cases and provided feedback for Noora to learn from.
An example of the NOORA interface and a sample of an interaction.
Putting Noora to the Test
To see how well Noora stacked up against treatment as usual, Koegel, Lam, and colleagues conducted a randomized trial with 30 participants in which half were assigned to use Noora for four weeks and half received no intervention. Participants using Noora were asked to complete 10 trials per day, five days a week for a total of 200 trials.
Ultimately, Koegel said, the team was looking to evaluate whether Noora could improve users’ empathetic responses and if improvements could be generalized to show empathy in human-to-human communication as well.
“There's a lot of AI research out there that shows that the ASD students improve using a program, but doesn’t show that it generalizes to real life,” said Koegel. “So that was our main goal.”
Comparing responses from the start of the experiment to the end, Koegel said that 71 percent of participants improved their number of empathetic responses when using Noora.
To see whether this progress could generalize, the team had participants take part in a Zoom call with a team member before and after the experiment, which included leading empathic statements. When reassessed after the intervention, the experimental group scored significantly higher than the control group with an average increase of 38 percent, while the control groups pre- and post-scores were similar. This shows that just four weeks of using the AI program significantly improved verbal empathetic responses.
With this success under Noora's belt, Koegel and Lam are now interested in testing the effectiveness of other modules as well. They’re also working to open Noora for beta testing for public use and in clinical settings.
Beyond Noora, Koegel said she’s also incorporating AI into other aspects of her autism research as well, including motivational treatment for children with ASD who are in the beginning stages of using communication.
“I'd like to take a lot of the work that I've done over the decades that's face-to-face and see how much we can translate to AI,” said Koegel. “Since kids really like the computer, we want to see if instead of just spending time on their computer or iPhone we can create a learning experience.”
The article:
A specialized chatbot named Noora is helping individuals with autism spectrum disorder practice their social skills on demand.
Knowing what to say when co-workers tell you about their weekend is a social skill that many take for granted, but for some individuals with autism spectrum disorder (ASD), this social interaction can be challenging. Struggling to find the correct response in social situations like these can negatively impact social and professional relationships for people with ASD, and can worsen co-occurring conditions like depression.
Research has shown that practicing social interactions with professionals in a clinical face-to-face intervention can improve outcomes for individuals, but these solutions are often costly or not widely available. Lynn Koegel, a clinical professor of psychiatry and behavioral sciences at Stanford University, in collaboration with Professor Monica Lam from Stanford’s Computer Science Department, are the authors of recent research published by the Journal of Autism and Developmental Disorders that investigates the role of AI in filling this gap.
“Our research has shown that face-to-face work does help with social conversation … so we wanted to see if we could translate that into computer use,” said Koegel. “Accessibility is really important because a lot of people don't have access to a face-to-face provider and the providers can be really expensive.”
Introducing Noora In this work, funded in part by a seed grant from the Stanford Institute for Human-Centered AI and the Kind World Foundation, Koegel and colleagues evaluated interactions between participants with ASD who struggle with social interactions and Noora, a chatbot built with a large language model (LLM).
In one-on-one interactions, which can be written or spoken, Noora offers individualized guidance on a number of social communication scenarios; it helps users learn to ask questions, give compliments, respond empathically, and with other areas of social communication that are often challenging.
In this recent work, Koegel focused on the impact of Noora's empathy module. The chatbot first offers a leading statement, such as “I’m feeling really tired lately and it’s been so hard to concentrate,” and then asks the user to assess whether the statement is positive, neutral, or negative. Noora will then grade this response and ask the user to respond empathically to the initial statement. Based on whether the user successfully responds with empathy, Noora either offers a gentle correction or validates a correct response.
The research team carefully crafted prompts with representative examples to ensure that the answers are appropriate. To interact with users, Noora needed to know three things: what kind of statements warrant an empathic response, how to assess whether a user has responded empathetically, and how to offer a user helpful feedback to improve the response if it lacked empathy.
To craft leading statements, the team exposed Noora to “golden” responses and to inappropriate responses. The team both wrote responses themselves and used the LLM to write other responses that they then verified, creating a pool of 330 statements designed to elicit empathetic responses from participants. This means that Noora was never creating leading statements on the fly, which could have potentially led to inappropriate questions.
When it came to responding live to users' empathetic responses, Noora had freer rein. Taking advantage of the LLM’s abilities for in-context learning, the team simulated users' personalities and had Noora practice responding to users that showed varying levels of empathy. They also selected difficult cases and provided feedback for Noora to learn from.
An example of the NOORA interface and a sample of an interaction.
Putting Noora to the Test To see how well Noora stacked up against treatment as usual, Koegel, Lam, and colleagues conducted a randomized trial with 30 participants in which half were assigned to use Noora for four weeks and half received no intervention. Participants using Noora were asked to complete 10 trials per day, five days a week for a total of 200 trials.
Ultimately, Koegel said, the team was looking to evaluate whether Noora could improve users’ empathetic responses and if improvements could be generalized to show empathy in human-to-human communication as well.
“There's a lot of AI research out there that shows that the ASD students improve using a program, but doesn’t show that it generalizes to real life,” said Koegel. “So that was our main goal.”
Comparing responses from the start of the experiment to the end, Koegel said that 71 percent of participants improved their number of empathetic responses when using Noora.
To see whether this progress could generalize, the team had participants take part in a Zoom call with a team member before and after the experiment, which included leading empathic statements. When reassessed after the intervention, the experimental group scored significantly higher than the control group with an average increase of 38 percent, while the control groups pre- and post-scores were similar. This shows that just four weeks of using the AI program significantly improved verbal empathetic responses.
With this success under Noora's belt, Koegel and Lam are now interested in testing the effectiveness of other modules as well. They’re also working to open Noora for beta testing for public use and in clinical settings.
Beyond Noora, Koegel said she’s also incorporating AI into other aspects of her autism research as well, including motivational treatment for children with ASD who are in the beginning stages of using communication.
“I'd like to take a lot of the work that I've done over the decades that's face-to-face and see how much we can translate to AI,” said Koegel. “Since kids really like the computer, we want to see if instead of just spending time on their computer or iPhone we can create a learning experience.”