What Makes Great Hand Tracking: A Crucial Guide for OEMs
Posted; March 28, 2023
In recent years, we've seen some of the biggest names in tech, including Meta, Bytedance, and Apple, make strategic moves into the immersive sector. In addition to record high VR sales since the pandemic began, acquisitions and investment into R&D have expanded the market globally, while making it more competitive than ever.
Hardware is evolving rapidly to meet increased user expectations. Many new sensors of all types are being included into different head mounted displays (HMD), and hand tracking has become table stakes for all XR headsets. As an original equipment manufacturer (OEM), how do you make the right decisions on the best technology to use for your product?
What is Hand Tracking and Why is it Important to OEMs?
Effective hand tracking allows for the most natural interaction between the user and the technology. The use of our hands has taken millennia to evolve into what it is today. We are immediately comfortable using our hands. Give an iPad to a toddler and see how easily they choose their favourite app. Our hands are capable of complex gestures and understanding their movements across a diverse population isn't an easy task.
A single gesture can change the entire tone of an interaction. Just as a child instinctively reaches out to touch something that looks interesting, engaging with technology as instinctively as using one's own hands should be at the heart of hand tracking.
Hand tracking has the capability to bring the benefits of VR and AR to a much broader audience and the responsibility to account for diversity. Human hands come in all shapes, colours and sizes. As we age, the form of our hands and fingers can also change. Asking for technology to understand this kind of subtle, nuanced exchange is asking a lot, but enabling more people to use their hands as they naturally would, rather than using controllers will expand user base tremendously.
When it comes to the technology itself, users tend to have a very low tolerance for faults. In a world where we expect technology to integrate with our lives seamlessly, we are quick to give up on ineffective machinery.
At Ultraleap, we are working with the benefit of more than a decade of development and iteration. Generations of research in AI. The feedback of hundreds of thousands of developers ensures our hand tracking just works, as it should. Just as expected.
Three Key Elements
Hand tracking can play a crucial role in applications with a vast array of different use cases. Training an airplane cabin crew to build muscle memory for safety procedures in a simulated environment requires a very different style of hand tracking from a family playing in a VR arcade ride.
The three key elements of world-class hand tracking are reliability, speed and accuracy. Each will vary depending on the use case. The one thing all users have in common is that tolerance for learning curves and bugs is low.
Anyone considering hand tracking technology should be asking questions based on these three elements:
1. Speed
Ultraleap hand tracking offers the option for a large 170 x 170 degree field of view and high frame rate. These capabilities, coupled with specifically trained machine learning, allows for your hands to be tracked before you can even seem them. For example, the Varjo XR-3 display has an industry leading 115 degree wide FOV. With the high frame rate, you get superb low latency tracking throughout the large volume of space created, even at high speeds.
Whether you are reacting to a fast in-game action or attempting to navigate product information at the required rate in a training app, Ultraleap’s technology offers lightning-quick reactivity.
2. Reliability
Any reliability issues in hand tracking would make people give up on the first hurdle. Hand tracking needs to work for people in diverse conditions and environments. Tracking the movement of hands against a single colour background is a very different challenge than in a busy environment.
Users shouldn’t have to change their behaviour or their natural environment for the technology to work. Our hand tracking technology works across all environmental conditions and complex backgrounds. It is also trained to respond to diverse hands.
3. Accuracy
Accuracy is essential. We work with 26 distinct data points on the hand, including bones and joints, which we track for improved usability. In many cases, they're usable even when one hand obscures the other. In enterprise environments like training simulations, users need to interact with virtual cockpits in XR as they would in the real world. A true simulation is not possible without precision.
Accuracy paves the way for natural movement which is important as we move into the future of people operating within virtual worlds. Digital version of people must be able to move their hands and arms just like their human counterparts. Mark Zuckerberg recently claimed that ‘legs are hard’; hands and arms are, arguably, even harder. Our technology also tracks a 27th bone, the elbow, which is critical for providing natural movements in relation to those of the hand.
How Do You Test Hand Tracking?
It’s vital that anyone testing out a developer kit from any hand tracking provider probes the system to see what it can achieve. Think about your use case(s) or your end user use cases. How you’re going to use hand tracking should determine where you focus your attention. For a gaming application, the focus should be on reliability and speed, precision may be less important.
We recommend testing hand tracking with the following elements in mind:
- Variation of conditions
Try it in low light, cluttered backgrounds, daylight and artificial light. Performance should be consistent across the board. You can read more about how our infrared cameras offer the best experience here.
- Complex range of hand moves
Try interacting with your hands close together, or with interlocking fingers to really test the capability. Try fast hand movements and precise gestures like a finger pinch. This will test accuracy as well as give you an impression of how ‘human’ a simulated hand looks on screen.
- Power consumption
Test the power input vs the output. There is a drive across the industry to reduce the power needed to get hand tracking working. Our technology employs significantly lower power use.
- Initialisation speed, latency and field of view
Ensure the time frame works for you and your needs. How long does it take to get your hands in view? Are there any noticeable delays?
- A wide range of different users
Test the technology out with different people. Knowing it can cope with diverse hands is crucial.
Our hand tracking technology meets the most demanding quality measures. We’re always happy to invite our customers to put our tech through a robust road test. Our team consists of some of the most experienced experts in the world, supported by a passionate, expert developer community. We never stop adjusting, improving, and pushing the boundaries of what’s possible.
Backed by a decade of hand tracking thought leadership we are building the future of human technology. Want to find out more? Get in touch we’d love to talk.
Explore our blogs, whitepapers and case studies to find out more.
