Immersive Real-Time Language Translator for Augmented Reality

Real-time translation systems have become widely available on traditional platforms such as smartphones and desktop applications. While smartphones and human interpreters provide rapid translation, these solutions are obtrusive and disrupt natural conversational flow. Augmented Reality (AR) glasses offer a promising alternative by embedding translation directly in the user's real environment. However, how different translation modalities should be presented in AR and how they affect comprehension in face-to-face conversation remains underexplored.In this thesis, we present a real-time multilingual communication system implemented on Snap's Spectacles AR glasses, which integrates automatic speech recognition (ASR), large language model (LLM)-based translation and text-to-speech (TTS) synthesis. The system enables two users speaking different languages with speech-to-speech translation returning under four seconds. Three translation modalities were designed and evaluated: (1) subtitles, (2) synthetic speech (TTS) and (3) a multimodal combination of subtitles and synthetic speech.In order to evaluate the system and the effect of translation modalities, a within-subjects user study (N=24) was conducted, in which 12 pairs experienced all three conditions while holding a conversation in two distinct languages. Conversational comprehension, translation quality, interaction quality, usability and cognitive load were assessed through questionnaires, alongside objective measures for task and system performance.Results show that translation modality affected perceived conversational comprehension. Participants reported that their conversation partners responded more appropriately when using synthetic speech or the multimodal condition compared to subtitles alone. Perceived task success was also higher in the multimodal condition, despite no significant differences in objective task performance. Based on these findings, design recommendations for future AR translation systems are proposed, which address latency, subtitle pacing and modality synchronization. Potential applications of AR-assisted multilingual communication are discussed across domains, such as language learning, education, healthcare and accessibility.We demonstrate that AR glasses can effectively support real-time multilingual communication, offering a practical alternative to traditional translation approaches.