Student computer science, mathematics, optical engineering or similar (f/m/x)

Digital Twins can be a powerful tool to raise the situational awareness of emergency forces and operatives of critical infrastructures in crisis situations and therefore support during decision making. One approach to achieve this is the provision of an interactive virtual reality environment representing the current situation via a remote service. However, to achieve this, there is an urgent need for a realistic incorporation of humans into the virtual environment in real-time. With the concept of MetaHumans, the Unreal Engine provides a promising approach for a realistic visualization, while tools like OpenPose allow translation of camera footage into a virtual skeleton. Thus, combining these approaches may enable a realistic real-time visualization of humans within the VR environment using a pipeline that translates camera footage into a metahuman.

The objective of this work is the development of a pipeline that extracts humans from camera footage and transfers them into Unreal Engine MetaHumans in near real-time. This includes the AI-based extraction of the location and pose of a human from camera footage using libraries such as OpenPose, the generation of a corresponding MetaHuman in the Unreal Engine and the transfer of the extracted pose into its skeleton.

• evaluating existing libraries for pose estimation with respect to skeleton reconstruction and spatial localization from camera footage
• developing a pipeline for the generation of MetaHumans in the Unreal Engine based on external data sources