Practicum topic

The main task in this practicum project will be to extend the existing Unity implementation of conversational agents to enable two new features:

  1. Push to talk button – currently the agents are always listening to nearby users no matter if they speak to them or to other people in the room. Therefore, a push to talk functionality needs to be implemented
  2. Integration of existing scene information into natural language understanding (NLU) call – In current implementation the information is obtained from Unity scene using a HTTP request to unity from external NLU server. This back-connection needs to be avoided and therefore we need to send all scene information in NLU request and filter it on NLU server side.

The existing implementation is fully functional and it offers a student to dive into the system of embodied conversational agents in VR and extend it. Additionally, the reward for successful implementation will be 500 €.

The current system uses Unity to create virtual character and to integrate it into virtual reality. Automatic speech recognition and text to speech are realized using NVIDIA Riva. Natural language understanding is implemented using customized Rasa server. Currently, the implementation also contains simple character animations and lip-sync.

In the future, the topic can be extended to a Master thesis by adding advanced functionality to the agents (e.g. advanced real-time generated animations, emotions, autonomous decisions of agents, etc.)

 

Requirements

Knowledge of Unity game engine and C# programming

 

Responsible

For more information please contact Peter Kán.