Tuesday 28 February 2012

Gesture Controls for Natural User Interfaces

I've been working on a few projects and I have returned to my Windows Media Player Natural User Interface. The newest version is very much a learning experience, and while it is currently fairly untested, there is much I can share. I thought I would start with my thoughts on hand gestures for my media player.

My first thoughts was to greatly limit the controls available for hand gestures to allow it to be as simple as possible. I decided on play/pause, fast forward, rewind, volume up and down. To turn on the "virtual remote" the user would raise their hand to about shoulder height, and press forward. Once it is activated, the coloured button would be the selected button and the user could press forward to activate it's control. To remove the remote, the hand would be lowered.

Originally I decided to control the remote similar to a mouse. The hand would essentially be a pointer, and you'd hover over a button and select. This quickly showed to be difficult. A major issue was that people would tend to push with their elbow causing the hand to lower and the target below, or no target, to be selected. I quickly decided to opt out of the "mouse-like" controls.

I've named the method we use in the newer version a "bump" style selection. This meant that a user could select  a button by bumping their hand in which ever direction. This removed the possibility of selecting the incorrect button.

With more testing we noticed a few more issues. Lowering the hand would select volume down always, and people would like to control volume up and down more easily, as well as their was excessive motions when selecting fast forward or rewind.



I decided that no one would select fast forward or rewind without wanting to press it. To use fast forward, a user would have to select fast forward, press fast forward, select play, press play. It seemed to be easier to just allow the user to select fast forward, immediately activate control of fast forward and move the selection to play. Now to fast forward a user would bump their hand to the right, then press forward to play again, reducing the amount of controls by 50%.

Finally, I made the volume controls a slider. A user bumps up, moves hand left and right, and bumps down.

These controls are incredibly slick and easy to use. The newest issue is people have a hard time wanting to keep their hand raised while using the controls and will lower them constantly while deciding which button to use. But that will be for another time.

(The newest version has also been ported over to WPF Applications for more slick design)

Anyone interested in the code for the remote can see it here