This Unreal Engine project was the culmination of much of my work in motion design and XR. This environment combined an Unreal Engine scene, Disguise Media Server rendering, Kinect tracking through TouchDesigner, physical lighting, and Red Eye Camera Tracking to create an interactable and immersive scene. I was responsible for the design and creation of the scene in Unreal Engine and the integration of the Kinect motion tracking data into Unreal Engine.
My primary objective when concepting this scene was to utilize Disguise and Unreal Engine's front and back plate functionality to create an object that could pass around a user, not just remain static in front of or behind them. To accomplish this, the user's position is tracked through a Kinect sensor, and their location is converted into coordinates in the Unreal Engine scene. The "will o' the wisp" then moves around those coordinates to ensure that it follows the user around the scene. If the wisp is closer to the camera than the user's coordinates, only the front plate version of the wisp is shown, while if it is further only the backplate version is shown.
The scene extends in 360 degrees, and we blend the edge of the physical screen with the virtual camera's view of the digital scene in order to achieve a seamless transition when the camera leaves the view of the stage.
Enormous thanks to the team that worked on this project with me - Nathan Mysona, who assisted me in integrating the scene into Disguise, JJ Garvey, who set up the physical lighting to match the digital scene lighting, and Angel Hardy, who set up the TouchDesigner motion tracking that would later be passed through into Unreal Engine.
Scene assets sourced from Megascans and "Deep Elder Caves" on the Unreal Engine Marketplace.