Abstract
This thesis presents a pipeline that leverages real-world data, such as point clouds and images, to create digital twins for autonomous driving simulations. Simulations play a crucial role in the development of safe automated driving systems, as they enable cost-effective and risk-free testing. To ensure the reliability of these simulations, virtual environments must closely resemble the real world. Our pipeline generates high-quality 3D meshes with photorealistic textures from the input data. Additionally, a 3D semantic segmentation of the reconstructed mesh is performed, providing ground truth data for downstream simulation tasks. This semantic segmentation is achieved using a virtual-view approach, where 2D renderings of the scene are segmented with an off-the-shelf model, and the predictions are projected back into the 3D scene. Information about the road layout and lanes is obtained from OpenStreetMap and aligned with the mesh. Finally, the pipeline output is used to create a virtual map in the driving simulator CARLA. We captured image and point cloud data from three locations and tested the pipeline using this input. We compared the differences in reconstructions from both input modalities, assessed their feasibility, and evaluated their effectiveness for practical applications. Reconstructions of the scenes were manually semantically annotated to provide ground truth for quantitative evaluation of the 3D semantic segmentation algorithm. The pipeline was implemented in Python with the goal of achieving a high degree of automation. It can produce a high-quality digital twin in a matter of hours, requiring minimal user intervention of under 20 minutes. The semantic segmentation algorithm achieves an mIoU of 55.2 and an F1 score of 67.1, reflecting a good performance for labeling the vertices of our datasets. This streamlined approach is a step forward for safer and faster development of automated driving systems.
Reference
Schenzel, G. (2024). Automated Digital Content Creation from Point Clouds and Image Data [Diploma Thesis, Technische Universität Wien]. reposiTUm. https://doi.org/10.34726/hss.2024.123826