Under the hood of any quality virtual, augmented, or mixed reality head-mounted display (HMD) is a hand and eye tracking platform designed to run a human-computer interface (HCI) capable of seamlessly joining man with machine and delivering seamless immersive experiences.
Creating hand and eye tracking solutions is a challenging task, namely as companies are challenged with bridging the gap between the physical and digital, both for content creators and the hardware manufacturers used to offer such experiences.
For our XR Today round table, we are pleased to welcome:
- Johan Hellqvist, Head of XR Segment at Tobii
- Ben Cathcart, Product Marketing Manager at Varjo Technologies
Our panellists represent some of the top companies in the industry for hand and eye tracking, and their technologies are leading the entire sector to monumental progress as they build solutions for developing the consumer and enterprise metaverses.
XR Today: Can you explain your hand and eye tracking solutions, and how they differ from others?
Ben Cathcart: Varjo provides highly accurate and proprietary eye tracking across its entire range of headsets, using hand tracking tools from Ultraleap’s fifth-generation Gemini technology available on our flagship XR-3 and VR-3 enterprise headsets.
Varjo’s eye tracking technology is video-based, leveraging two built-in, high-speed cameras and infrared illumination to capture images of a user’s eyes at 200 hertz while measuring information such as pupil position, pupil dilation, interpupillary distance (IPD), focus or fixations, eye-motion patterns and more.
Varjo’s eye tracking brings improved graphics performance for foveated rendering, which reduces unnecessary pixels in the users’ peripheral view, and is a key component in Varjo’s renowned human-eye resolution bionic displays.
Ben Cathcart, Product Marketing Manager at Varjo Technologies
Eye-tracking data can also be monitored in real-time and exported for multiple research use cases in medical, usability, marketing, psychology, user behaviour studies, and more.
New use cases for eye tracking data are also consistently being used, such as more naturally replicating eye movement on digital avatars in multi-user virtual or mixed reality (MR) experiences.
Varjo’s eye tracking solution is highly accurate compared to others due to its unique use of infrared illumination patterns, allowing for more precise and consistent results even with the use of eyeglasses.
Varjo’s XR-3 and VR-3 headsets were the first VR and XR headsets to incorporate Ultraleap’s highly accurate 5th-gen Gemini hand tracking, which features improved initialization, pose accuracy, and occlusion handling than ever before.
Also, using a combination of cameras and infrared LEDs, the XR-3 and VR-3 headsets track the position of the hands, allowing for virtual hand models to render in a VR and XR scene in real-time.
With hand tracking, users can intuitively interact with virtual objects, deepen the realism and immersion of simulations, and use gestures to navigate VR and XR software to communicate better with more lifelike human avatars.
Johan Hellqvist: Regarding eye tracking, we believe the technology is essential and foundational [to XR]. It equips headsets with the capability of monitoring user attention in real-time. This information can be used to measure a range of physiological data, allowing for better interactions between machines and their users.
Tobii is the industry’s most experienced independent provider of eye tracking technology, and we have more experience running eye tracking in XR than any other player out there, resulting in the most robust technology available today.
If you are targeting user population coverage, you must consider scenarios with people once considered outliers as now integral cases that need to be solved. For example, cases with droopy eyelids, makeup covering vital features, prescription glasses, contact lenses, and lazy or dominant eyes must be solved. Additionally, users face headset slippage, variations in interpupillary distance, face shapes, skin reflectance in near-infrared, iris colour and components, and placement tolerances, among many others.
XR Today: Why is seamless hand and eye tracking essential to a flagship (or any) headset?
Ben Cathcart: Apart from its wide range of research use cases and implementation in multi-user social VR and XR experiences, eye tracking is essential for high visual fidelity VR/MR experiences, as it is a fundamental component of foveated rendering. The industry-leading visual fidelity provided across Varjo’s entire hardware lineup simply would not be possible without it.
Whilst not always essential, for example for users with relevant physical handicaps or in simulation scenarios that focus on physical peripheral usage such as joystick control within flight simulators such as Microsoft Flight Simulator 2020. Hand-tracking is highly beneficial for use cases featuring intuitive hand-based interactions like personnel training in industrial, aviation, and healthcare simulations.
Compared to using physical controllers, the barrier of entry is also significantly reduced, especially for first-time users, as they can use their hands in an entirely intuitive and natural input method.
Johan Hellqvist: Eye tracking is undoubtedly essential for headsets. The aim of VR is ultimate immersion, where users can forget where they are and move in multiple other spaces where reality and simulation seamlessly combine.
Achieving that requires VR applications with original features and good storytelling delivered on high-performance headsets that can balance load across subsystem components to achieve the best possible user experience. Eye tracking is a core component needed in VR headsets to strike that balance.
Johan Hellqvist, Head of XR Segment at Tobii
Significant developments have occurred in VR over the past few years, notwithstanding the emergence of enterprise-grade applications and standalone headsets. Specifically, development in display technologies has created a demand for higher screen resolutions, pixel densities, and refresh rates with expanding field of view (FoV) capabilities to improve the visual experience. Unfortunately, this comes at the cost of processing resources.
Frame rates of 90Hz are becoming the gold standard for VR headsets, which help to reduce motion sickness and headaches that can occur at lower frame rates. However, frame rates are not just about what a headset can support, but maintaining a stable frame rate for a given application is fundamental to an immersive experience — eye tracking is needed for this.
At the same time, as eye tracking technology is included in headsets, it is also made available more directly to developers, allowing them to create increasingly natural and immersive experiences and evaluate user experience and usability.
XR Today: Could you explain some of the most common use cases for your product? How can firms apply these technologies across sectors?
Ben Cathcart: Varjo devices are built for visionaries in some of the world’s most demanding industries like training and simulation, design, architecture, engineering, and construction, medical, marketing, as well as sales, product design reviews, behavioural research, and others, providing the highest visual fidelity available today for at-home VR enthusiasts of flight and racing simulations.
Today, 25 percent of Fortune 100 companies use Varjo’s XR/VR headsets, which deliver unprecedented return on investment (ROI) to the world’s biggest brands. Varjo features multiple detailed commercial customer stories on its website, covering astronaut training with Boeing, automotive design with Kia and Volvo, pilot training with VRM Switzerland, marketing with MUJI, and many more.
With Varjo, the Boeing Starliner programme unlocks an entirely new way for astronauts to prepare for spaceflight. Varjo facilitates astronaut training, from pre-launch to docking to landing, entirely in VR for the first time.
Together, Volvo Cars and Varjo are driving the future by creating the world’s first MR approach to evaluating prototypes, designs and active safety technologies. Both companies have made it possible to drive a real car while wearing a headset while seamlessly adding realistic, lifelike virtual elements.
Siemens Corporate Technology is also democratizing product development by moving it to a virtual environment and has chosen Varjo for their VR device of choice.
OpenBCI, a Brooklyn-based neurotechnology company, has also partnered with us for the launch of Galea, its newest product. Galea’s beta system has been integrated with the Varjo Aero headset and is the world’s first device that simultaneously measures the user’s heart, skin, muscles, eyes, and brain.
The Galea Beta Program recently received significant interest from applicants across consumer technology, healthcare, research, training, gaming, and interactive media.
With Varjo’s human-eye resolution VR/MR technologies, learners can take full advantage of Toltech’s real anatomical imaging to observe and interact with the human body in photorealistic 3D.
The cost of physical mockups can run up to a million dollars. With Varjo, it’s now possible to replace physical mockups with virtual ones. Sellen Construction saves time, money and effort throughout the entire lifecycle of a building project with VR.
Japanese retail company MUJI have also harnessed VR to achieve a deeper connection with their in-store customers, and the Varjo Aero enables crystal-clear communication of their core values.
Johan Hellqvist: We have learned that eye tracking empowers devices through dynamic foveated rendering, which is a unique performance-enhancing technology that can significantly reduce rendering loads in VR headsets.
In practice, this technology relies on eye-tracking to deliver the necessary real-time data of a user’s eye movements. It then enables better computing efficiency and display performance by rendering in higher resolution only where you are actually looking.
Eye-tracking helps users to handle VR headsets properly by using display positional tools. These leverage eye-tracking and information on the positioning of a user’s relative head display and lenses, which ensure users can adjust the headset to an optimal position.
Eye tracking provides headsets with a way to detect user attention, leading to improved and greater natural experiences. Many interfaces and 3D experiences benefit highly by adding eye tracking to select objects to manoeuvre VR environments.
Animating avatars based on users’ eye movements also bring characters to life in a way that cannot be overstated. For example, characters that blink and whose eyes dart around are much more believable than others. Eye contact is essential in human communication. If my counterpart looks into my eyes, to the side, or at my lips, each has a different social meaning.
The eyes are also said to be the window to the soul. By measuring and analyzing user gaze and attention in training situations, medical diagnostics, or user-centric software, developers give practitioners a fantastic tool to qualitatively understand user reactions and quantitatively measure human behaviour.
XR Today: What future capabilities do you envision for hand and eye tracking? Will they eliminate the need for physical controllers, or potentially monitor psychological responses to certain stimuli?
Ben Cathcart: Dedicated teams at Varjo and Ultraleap are working consistently to raise the already high bar for accuracy and reliability of our inbuilt eye and hand-tracking solutions.
This ensures that all variations of the Metaverse experienced with Varjo’s technologies are entirely seamless, accessible, and at the absolute forefront of visual fidelity. Tracking how people react while interacting with VR will improve immersion through two specific methods.
The potential uses already derived from existing hand and eye tracking data have only partially been tapped. We can expect this to increase as software offerings evolve and XR technology becomes more ubiquitous in both our professional and day-to-day lives.
Johan Hellqvist: As eye tracking moves into consumer VR, I expect to see it evolve on three fronts. Firstly, the technology has become smaller in size, more optimized for power consumption, and with more extensive adoption globally.
Secondly, Eye tracking use cases will be honed by the ecosystem, and I think we will start seeing very broad usage of its capabilities, both for device enhancements such as foveated rendering and for user interactions
Finally, I expect us to see improved and expanded capabilities such as enhancing performance for technologies leveraging eye tracking, namely foveated rendering, social interactions, and analytics.
