Abstract

Mixed reality, in combination with virtual production, offers the potential to present and include mixed real-digital 3D content in applications such as live online events, journalism, entertainment and education. However, its potential has not yet been fully exploited: virtual production is traditionally 2D and possibilities for interacting with 3D data video streams are limited. Additionally, the options for camera motion in virtual production are often limited by the size of a background LED wall. VRStream addresses the challenges of 3D virtual production in a live-streamed format, real-time interaction of users with the content in virtual reality, and advanced camera control enabling full 360 degrees virtual camera panning even with a limited LED wall size.

VRStream is set to achieve the following research goals:

  • A novel method for 3D virtual production using synchronized stereo cameras and an LED wall
  • Advanced interface for camera control using rotational gains
  • Real-time interaction of an actor with digital content using marker-less motion capture
  • A method for multi-content 3D live streaming
  • Novel interfaces for user interaction with the received 3D digital content in VR
  • Gaining new knowledge about acceptance and usability of the proposed methodologies for interaction with live mixed reality content

Hardware

LED Wall at the Mixed Reality Lab

Project partners

  • Cyberith GmbH
  • Take Two Studio OG

Funding provided by

  • FFG - Österr. Forschungsförderungs- gesellschaft mbH