SynVis: Digitising Grapheme-Colour Synaesthesia Through Augmented Reality

The most common type of synaesthesia, known as grapheme-colour synaesthesia, causes unique sensations in which letters and numbers are associated with specific colours. With the rapid expansion of technology, particularly in the field of assistive technology, this Master’s thesis investigates the visual replication of grapheme-colour synaesthesia using Augmented Reality (AR).The goal of this thesis is to see if synaesthetes’ different individual perceptions could be coded into simple, machine-implementable rule-sets that would allow synaesthetes to pre-colour achromatic text before reading. Furthermore, it aims to establish the technical requirements for implementing such a system. Therefore, a literature review is conducted to find out if there are specific recurring patterns in how synaesthetes perceive colours on the word level. Based on previous research and expert interviews with a synaesthesia researcher and a synaesthete, the identification and formalisation of the regulatory factors that elicit specific colours in synaesthetes are validated. This allows for the creation of a prototype that uses mobile AR to represent grapheme-colour synaesthesia perceptions. The app enables the recolouring of real-world text using the device’s camera input, based on different rule-sets provided. To analyse this qualitatively and quantitatively, the thesis includes expert interviews, benchmark tests to assess the app performance (frame time (FT), response rate (RR), and error rate (ER)), and a user study to determine technological feasibility. Evaluating the various visualisation alternatives reveals a preference for minimum invasive visualisations.The most effective method is to outline text in the appropriate colours directly on the pixel basis of the camera picture. Visualising hues as backdrops behind black lettering,on the other hand, is disliked due to readability concerns with dark colours. The work addresses technical hurdles and suggests options for future research, opening the door for more research in this area.