International Conference on Computer and Knowledge Engineering

Home / 14th International Conference on Computer and Knowledge Engineering

Automating Theory of Mind Assessment with a LLaMA-3-Powered Chatbot: Enhancing Faux Pas Detection in Autism

Authors :

Avisa Fallah¹ Ali Keramati² Mohammad Ali Nazari³ Fatemeh Sadat Mirfazeli⁴

1- School of Mathematics, Statistics and Computer Science, University of Tehran,Tehran, Iran 2- School of Electrical and Computer Engineering University of Tehran,Tehran, Iran 3- Department of Neuroscience, Advanced Technologies in Medicine, Iran University of Medical Sciences, Tehran, Iran 4- National Brain Centre, Iran University of Medical Sciences, Tehran, Iran

Keywords :

Large Language Models،Interactive Chatbot،Faux Pas Recognition Test،Chain-of-thought،Theory of Mind

Abstract :

The Theory of Mind (ToM) is critical for social interactions, enabling individuals to understand and predict others’ mental states. Deficits in ToM, particularly in detecting faux pas, are common among individuals with Autism Spectrum Disorder (ASD). Traditional methods for assessing ToM, such as the Faux Pas Recognition Test (FPRT), require significant human involvement, which can limit accessibility and scalability. This study aims to explore how large language model (LLM)-powered chatbots, specifically utilizing the LLaMA-3 model, can enhance the assessment of ToM by automating and improving the administration of the FPRT. The methodology involved creating a dual-phase process where the chatbot first assesses user responses and then provides rehabilitative guidance to correct misunderstandings. Additionally, the system employs a scoring mechanism combining automated evaluation with human oversight to ensure accuracy and consistency, with system prompts crafted based on chain-of-thought prompting to enhance the depth and reliability of the assessments. Our findings demonstrate the effectiveness of LLM-powered chatbots in delivering dynamic and supportive ToM assessments, suggesting its potential for broader application for both artificial intelligence and psychological practices.