New Tech Makes Mixed Reality More Interactive

• By Dian Schaffhauser
• 06/14/18

With the mixed reality solution, a simulated world can be combined with the real world in real time. Source: Fraunhofer HHI

In the future, according to a group of German researchers, users of virtual reality will be able to interact with each other more easily, more naturally and in real time between real and virtual worlds. A scene from the real world could, for example, be transferred into VR, after which feedback could be reflected back into the real situation. The Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (HHI) recently presented a new "X-reality technology" that makes this possible; the demonstration is being shown during Germany's CeBIT conference, taking place this week. Fraunhofer HHI does applied research in mobile and optical wireless communications and video signal processing and coding.

The X-reality process has two sides to it, neither of which involves the clunky controllers that are now common in VR usage.

In the "real world," the scene is recorded from all sides by eight 3D cameras, and then converted into "depth maps." The cameras also monitor gestures and dynamic movements. That stream of data is put through a set of algorithms developed by the institute, coded and transmitted to a VR station in real time with the corresponding 3D textures.

In the virtual scenario, another 3D camera records the VR user. In the scene, the user is represented as a moving, full-body avatar; in other words, the user is seeing his or her own body and gestures in the virtual space.

That combination of technologies produces a kind of mixed-reality interaction that facilitates "collaboration scenarios," explained Project Manager Paul Chojecki, in a press release. Projection allows feedback from the virtual world to be displayed in the real scene. Image processing algorithms, also from Fraunhofer HHI, ensure that elements are displayed in an "optically correct" way, even if the surfaces on which they are projected are tilted or move.

The institute suggested that the applications for X-Reality are diverse — useful anywhere collaboration needs to happen at a distance. Examples include remote assistance, rapid prototyping, human-robot interaction, telepresence and, of course, gaming (where two people separated from each other could play a board game together). For instance, in the area of remote assistance, when a technician needs to repair something on site, somebody back in the office could get a virtual picture of the situation in 3D using VR glasses, and intervene where needed by demonstrating to the technician the correct components or hand movements.

"The solution we developed for this purpose can connect a simulated world with the real world in real time and high quality and open up new perspectives or collaborations," said Chojecki. "Physical interaction without disturbing controllers is more natural and more comfortable. The solution can be adapted more flexibly to the height of the user and increases immersion. At the same time, it can reduce symptoms of motion sickness that are often caused by VR scenarios."

During the conference, the institute is allowing visitors to try out a 3D puzzle and get help from the virtual world. In the demonstration, the real object is projected live into the VR environment and explained to the VR user. The user can then react virtually and have those reactions reflected back into the real situation.