Mixed Reality has become a disruptive force that is bridging the gap between reality and imagination. With AI, it is now poised to change the world as we see it!
The global mixed reality market is expected to reach USD 6.86 billion by 2024. Mixed Reality has found application in not just the obvious gaming and entertainment industries but also has great potential in business and other industries ranging from manufacturing, travel, and medicine to advertising. Maybe that is why the biggest names in tech are battling it out to capture the MR market with their devices – Microsoft HoloLens, GoogleGlass 2.0, Meta2 handsets to name a few. Incorporating Artificial Intelligence is their next step towards MR market domination.
So what’s all the hype about MR and how can AI take it to the next level?
Through the looking glass: Understanding Mixed Reality
Mixed reality is essentially a fantastic concoction of virtual reality (a virtual world with virtual objects) and augmented reality (the real world with digital information). This means virtual objects are overlaid in the real world and mixed reality enables the person who is experiencing the MR environment to perceive virtual objects as ”real ones”. While in augmented reality it is easy to break the illusion and recognize that the objects are not real (Hello Pokemon Go!), in Mixed Reality, it is harder to break the illusion as virtual objects behave like real-world objects. So when you lean in close or interact with a virtual object in MR, it gets closer in a way a real object would.
The MR experience is made possible with mixed reality devices that are typically lightweight and wearable. They are generally equipped with front-mounted cameras to recognize the distinctive features of the real world (such as objects and walls) and blend them with the virtual reality as seen through the headset. They also include a processor for processing the information relayed by an array of sensors embedded in the headset. These processors run the algorithms used for pattern recognition on a cloud-based server. These devices then use a projector for displaying virtual images in real environments which are finally reflected to the eye with the help of beam-splitting technology.
All this sounds magical already, what can AI do for MR to top this?
Curiouser and curiouser: The AI-powered Mixed Reality
Mixed Reality and Artificial Intelligence are two powerful technology tools. The convergence of the two means a seamless immersive experience for users that blends the virtual and physical reality.
Mixed Reality devices already enable interaction of virtual holograms in the physical environment and thereby combine virtual worlds with reality. But most MR devices require a large number of calculations and adjustments to accurately determine the position of a virtual object in a real-world scenario. They then apply rules and logic to those objects to make them behave like real-world objects. As these computations happen on the cloud, the results have perceivable time lag which comes in the way of giving the user a truly immersive experience. Also, user mobility is restricted due to current device limitations.
Recently there has been a rise of the AI coprocessor in Mixed Reality devices. The announcement of Microsoft’s HoloLens 2 project, an upgrade to the existing MR device which now includes an AI coprocessor is a case in point.
By using AI chips for computing, the above calculations, for example, MR devices will deliver high precision results faster. It means algorithms and calculations can run instantaneously without the need for data to be sent to/from a cloud. Having the data locally on your headset will eliminate time lag, thereby creating more real-time immersive experiences. In other words, as the visual data is analyzed directly on the device and computationally-exhaustive tasks are performed close to the data source, the enhanced processing speed results in quicker performance. Since the data remains on your headset always, fewer computations are needed to be performed on the cloud, hence the data is more secure.
Using an AI chip also allows flexible implementation of deep neural networks. They help in automating complex calculations such as depth perception estimations and generally provide a better understanding of the environment to the MR devices. Generative models from Deep Learning can be used to generate believable virtual characters (avatars) in the real world. Images can also be more intelligently compressed with AI techniques. This would enable faster transmission over wireless networks. Motion capture techniques are now employing AI functionalities such as phase-functioned neural networks and self-teaching AI. They use machine learning techniques to combine a vast library of stored movements and combine and fit them into new characters.
By using AI-powered Mixed Reality devices, the plan is to provide a more realistic experience which is fast and provides more mobility. The ultimate goal is to build AI-powered mixed reality devices that are intelligent and self-learning.
Follow the white rabbit: Applications of AI-power Mixed Reality
Let us look at various sectors where Artificially Intelligent Mixed Reality has started finding traction.
Gaming and Entertainment
In the field of gaming, procedural content generation techniques allow automatic generation of Mixed Reality games (as opposed to manual creation by game designers) by encoding elements such as individual structures, enemies etc with their relationships. Artificial Intelligence enhances PCG algorithms in object identification and in recognizing other relationships between the real and virtual objects. Deep learning techniques can be used for tasks like super resolution, photo to texture mapping, and texture multiplication.
Healthcare and Surgical Procedures
AI-powered mixed reality tech has also found its use in the field of Healthcare and surgical operations. Scopis has announced a mixed reality surgical navigation system that uses the Microsoft HoloLens for spinal surgery applications. It employs image recognition and manipulation techniques which allow the surgeon to see both the patient and a superimposed image of the pedicle screws (used for vertebrae fixation surgeries) for surgical procedures.
Retail
Retail is another sector which is under the spell of this AI infused MR tech. DigitalBridge, a mixed-reality company uses mixed reality, artificial intelligence, and deep learning to create a platform that allows consumers to virtually try on home decor products before buying them.
Image and Video Manipulation
AI algorithms and MR techniques can also enrich video and image manipulation. As we speak, Microsoft is readying the release of Microsoft Remix 3D service. This software adds “mixed reality” digital images and animations to videos. It keeps the digital content in the same position in relation to the real objects using image recognition, computer vision, and AI algorithms.
Military and Defence
AI-powered Mixed Reality is also finding use in the defense sector, where MR training simulations controlled by artificially intelligent software combine real people and physical environments with a virtual setup.
Construction and Homebuilding
Builders can visualize their options in life-sized models with MR devices. With AI, they can leave virtual messages or videos at key locations to keep other technicians and architects up to date when they’re away. Using MR and AI techniques, an architect can call a remote expert into the virtual environment if need be, and virtual assistants can be utilized for further assistance.
COINS is a construction and homebuilding organization which uses AI-powered Mixed Reality devices. They are collaborating with Sketchup for virtual messaging and 3D modeling and Microsoft for HoloLens and Skype Chatbot assistance.
Industrial Farming
Machine learning algorithms can be used to study a sensor-enabled field of crop and record the growth, requirements and anticipate future needs. An MR device can then provide a means to interact with the plants and analyze present conditions all the while adjusting future needs. Infosys’ Plant.IO is one such digital farm which when combined with the power of an MR device can overlay virtual objects over a real-world scenario.
Conclusion
Through these examples, we can see the rapid adoption of AI-powered Mixed Reality recipes across diverse fields enabled by the rise of AI chips and the employment of more exhaustive computations with complex machine learning and deep learning algorithms.
The next milestone is to see the rise of a Mixed Reality environment which is completely immersive and untethered. This would be made possible by the adoption of more complex AI techniques and advances made in the field of AI both in hardware and software. Instead of voice or search based commands in the MR environments, AI techniques will be used to harness eye and body gestures. As MR devices become smaller and mobile, AI-powered mixed reality will also give rise to intelligent application development incorporating mixed reality through the phone’s camera lens.
AI technologies would thus help expand the scope of MR not only as an interesting tool, with applications in gaming and entertainment, but also as a practical and useful approach for how we see, interact and learn from our environment.
