Hand simulation for virtual climbing

VR applications enable users to immerse into a virtual world. In current VR systems, users wear a Head-Mounted Display (HMD) and can use handheld controllers to interact with the virtual world. The controllers are suitable for a variety of applications, but they cannot be used if users are to interact with real objects. An example for such an application is the VreeClimber project. It combines a moveable climbing wall with VR climbing. Since the hands of the users should also be visible in the virtual world during climbing, an optical tracking system is used to capture the hand positions in real time. In the course of this thesis, two software components were created for the VreeClimber project. At first, the software of an already developed tracking system called VreeTracker was rewritten with the computer vision library OpenCV. During the development of this software, the affordable Vive Tracker was released, which can also be used to track extremities. The evaluation compares the accuracy of the two tracking systems by different tests. Since the users are not able to see their own hands during climbing, an algorithm was created in the second part of this thesis, which uses the detected hand positions to simulate the hand movements of the climbers. The developed hand simulation shows promising results for typical grasp movements during climbing, however especially fast movements or small climbing holds can result in deviations from the real hand pose. An important requirement is an accurate calibration before climbing, otherwise the positions of the real and virtual climbing holds may differ, which reduces the climbing experience significantly. The evaluation shows that the commercial Vive Tracker achieves better results than the previously developed VreeTracker system. Due to better precision and an easy integration into the already used VR system, it makes sense to use the Vive Tracker for the VreeClimber project in the future. In general, the virtual hand simulation performed well in the evaluation, however minor flaws in the grasp movements of individual fingers have been revealed. After a short analysis, appropriate suggestions for improvement have been presented in this thesis.