Design guide for gesture control in automotive HMI

Posted; May 7, 2020

Ultraleap’s Experience Design and Research team is building the standards for common gestures, interactions, and haptic sensations. Here, Matt Corrall, Design Director, shares our thinking around best practices for gesture control in automotive HMI.

The automotive HMI of the future is going to use a host of new technologies to respond more intelligently to a driver’s needs, increasing safety and letting you control your car in a way that’s natural and easy. It will likely be powered by facial and gesture recognition, voice control, visual displays, eye tracking, and haptic technology.

Ultraleap’s world-leading hand tracking powers reliable, customizable gesture control in automotive HMI – allowing drivers to control their car’s infotainment system with simple hand gestures.

Adding haptic feedback to gesture control using our “virtual touch” haptic technology gives tactile confirmation that an action has been completed, even though there’s no button or touchscreen present. Think of it as feeling a “force field” against your palm.

Our previous high-fidelity explorations proved that fewer glances away from the road occur when using gesture control augmented with mid-air haptics, compared with a traditional touchscreen interface.

However, gesture control is new enough that there are no established standards yet for how to do things well – unlike, say, designing for a touchscreen. Our team is building those standards for common gestures, interactions, and haptic sensations.

Here, we share our current thinking on best-practice gesture frameworks for automotive HMI.

In Ultraleap’s Experience Design and Research team, we study human behaviour and use evidence-led design practice to determine the gestures, interactions, and haptic sensations that work best. We work with customers integrating our technology into a wide range of products, using what we’ve learned to help them design intuitive, touchless experiences.

Gesture and interactions aren’t the same in every case. Our gesture choices will be affected by the context and job the user is doing, as well as whether they’re using a 2D screen or a form of virtual or augmented reality.

Are they a driver operating an infotainment system without taking their eyes off the road, or a vehicle designer reviewing a 3D model in VR? For each case, we start with the person and find the interactions that best fit their needs.

Recently, Groupe PSA’s new luxury concept car the DS Aero Sport Lounge featured Ultraleap’s hand tracking and mid-air haptic technology built into the futuristic car’s central armrest. Powered by Ultraleap, the driver (or passengers) can control a range of systems with natural hand gestures – from entertainment to navigation.

The gesture interactions in the DS Aero Sport Lounge concept car were based on insights from a previous Ultraleap infotainment system design project. These were investigated by designers in our Experience Design and Research team through iterative design, under representative workload conditions in a driving simulator.  

The scope of the design project was determined by the study of market-leading infotainment systems and qualitative user research with drivers from a wide range of backgrounds. This gave us a sense of how automotive HMI is commonly used and let us determine the most important functions.

Based on this, we designed a prototype gesture control infotainment system that allowed users to:

Our team designed and built a working prototype of the automotive HMI, conducting tests with 42 different users using a purpose-built driving simulator very similar to those used in the automotive industry. We then iterated upon the design and repeated our tests, learning which combination of hand gestures, haptic sensations, and on-screen UI components worked best together.

With gesture control, we need to consider the reliability with which the technology can recognise a gesture and interpret the user’s intention. Unlike pressing buttons on a touchscreen, hand gestures are analogue – my hand goes through a range of motions and postures as it forms a gesture and then releases it.

Hand shapes and sizes, the way people gesture and even ambient light will vary hugely, making it a technical challenge to get it right every time.

The hand tracking camera must also be mounted where it can get a good view of the user’s hands in the intended interaction area. This avoids occlusion (i.e. when it’s not clear what gesture you are making because part or all of your hand is blocked from the camera’s view).

During user testing, our team assesses hand gestures and associated haptics and UI elements in terms of their usability, their comfort, and their reliability.

The following are our current recommendations for using hand gestures in automotive HMI, each with suggested functions, pros and cons.

Picking the right gestures depends on many factors and must be considered as part of a wider design system that includes haptics, UI components, audio feedback, and vehicle architecture. Ultraleap are currently undergoing automotive grade validation of these elements. For now our guidelines are the best place to start.

Matt Corrall is Design Director at Ultraleap

Refers to how easy it is to learn, understand and remember the required form and movement of the gesture in order to perform it correctly. A gesture with high usability is considered easy to use relative to other gestures.

Refers to the likeliness of the hand tracking device correctly interpreting the gesture. A gesture with high reliability can be expected to be recognised the vast majority of the time.

Refers to how physically difficult a gesture is to perform, and how much discomfort is experienced whilst performing the gesture. A gesture with high comfort can be performed easily with minimal effort.

Using a flat hand, the driver “bounces” their whole hand down and up with a fast motion.

Examples of actions it could be used for in car HMIs:

Usability: Medium   |   Reliability: Medium   |   Comfort: High

Keeping their whole arm still, the driver moves their fingers (or a single finger) down and up with a fast motion. Requires less physical effort than the Hand Tap gesture.

Examples of actions it could be used for in car HMIs:

Usability: Medium   |   Reliability: Medium   |   Comfort: Medium

The driver brings their index fingertip and thumb together whilst keeping their remaining fingers open (exposing their palm to feel mid-air haptic feedback).

It is the same as the universal sign for “OK”, and can be either be held for adjusting values/objects, or performed quickly to serve as a mid-air confirmation action.

They can also move their hand up whilst performing the pinch gesture to increase a value (e.g. volume), and down to decrease the value, before opening their hand to confirm the choice.

This same gesture also works whilst moving in other directions, going left/right and forwards/backwards to control sliders, menus, and objects.

Examples of actions it could be used for in car HMIs:

Usability: Medium   |   Reliability: High   |   Comfort: High

The driver clenches their hand – making a fist - before opening it again. This is performed in one fluid motion.

Examples of actions it could be used for in car HMIs:

Usability: High   |   Reliability: Low   |   Comfort: High

The driver simply shows a certain number of fingers to the hand tracking device. This can be used – for example – as a shortcut to quickly move between infotainment modes like music, navigation etc.

Examples of actions it could be used for in car HMIs:

Usability: High   |   Reliability: Medium   |   Comfort: Low

The driver “flicks” their hand from left to right, then back again in one single motion. Only the forearm needs to move, rather than the whole arm.

Examples of actions it could be used for in car HMIs:

Usability: High   |   Reliability: Low   |   Comfort: Medium

Hand Twist

Starting with their hand open and palm facing down, the driver turns their hand over then back again in one fluid motion (hand rotates outwards rather than inwards).

Examples of actions it could be used for in car HMIs:

Usability: High   |   Reliability: High   |   Comfort: Medium

NOTE: The interactions outlined in this document have been designed and developed in line with user-centred design principles but have not been validated against relevant industry standards and guidelines (e.g. BS-EN-ISO 9241-500, BS-EN-ISO 15005, NHTSA-2010-0053, etc.). Any adoption of techniques explored in this document should be conducted in compliance with international automotive standards and in the context of the intended vehicle before implementation for use while driving.