Embodied Conversational Agents with Situation Awareness

Virtual worlds offer unlimited possibilities for creating educational training scenarios in various domains. These worlds are often enriched with embodied agents to simulate human behavior in various interactions between human users and virtual agents. However, these agents usually only have limited knowledge and behavior and their communication skills are usually predefined or they are not able to communicate at all. In this thesis, we investigate the impact of embodied agents with conversational abilities and situation awareness in a first responder training scenario on human perception and performance. We present a novel solution to enabling situation awareness for embodied agents which allows them to capture and react to changes in their environment and their own state. The agents are capable of conveying this captured knowledge through full conversational capabilities by utilizing a combination of novel methods from NVIDIA for automatic speech recognition and speech synthesis and the industry proven conversational Artificial Intelligence (AI) Rasa. To evaluate our conversational agents, we conducted a between-groups user study with 24 participants in a Virtual Reality (VR) training application in the Unity game engine and investigated the differences between agents with full conversational capabilities and agents with scripted audio. During the study we measured several quantitative metrics including presence, co-presence, task performance, realism, learning outcome, information presentation, agents interaction and training duration as well as qualitative measurements in the form of open questions. While our quantitative results did not indicate significant differences in all measured metrics, we found a significant difference in favor of agents with full conversational capabilities in the metric co-presence. In addition, we discovered significant differences between genders in the metrics subjective task performance and training duration. Finally, we discussed user feedback on our conversational enabled agents and derived guidelines for future research and development of training applications with embodied conversational agents with situation awareness in VR from our qualitative results.