Controlling VR Hands in 3D via Gamepad: Automatic Height Adjustment

Providing an alternative gamepad interaction approach, for otherwise spatially tracked hand controllers in VR, is not a trivial endeavor, but doable.

Due to the analogue sticks of a gamepad providing only two-dimensional inputs, a special approach was needed to three-dimensionally control the hands in Gooze with such an input device. The respective input approach is based on the player controlling the translation of a hand via an analogue stick on the X-Z plane only and using an algorithm to automatically adjust the hands Y position according to grabbable objects. So, the player can actively move his virtual hands forwards/backwards and sideways in a certain range, but the algorithm adjusts the upwards/downwards movement automatically, if grabbable objects are in range.

The latter is not a trivial behavior though, if it should feel rather unnoticeable and natural to the player. To achieve this feature the corresponding algorithm is based on several phases, which need to be rerun on every frame, as the three-dimensional constellation of objects and player hands can constantly change. In turn the corresponding code needed to be specifically optimized to run as performance efficient as possible.

In the first phase the system checks if and which grabbable objects are in grabbing range (see cups and polaroids in truncated light blue box in the image). Next, instead of just using the center points of the objects for further calculations, the two “optimal” grab points (one for each hand) are calculated for each object. This means the closest point on an object Collider (see green and blue cubes around objects in the image, green for left and blue for right hand) to the initial hand position (see isolated smaller green and blue cubes) is acquired. Including the current hand positions and the four corner points of the grab range box positioned at the height of the initial hand positions, this three-dimensional constellation of hand positions and “aim points” is then cached. For further calculations, it is flattened to two dimensions on the X-Z plane. Next, the space around each hand’s current position is separated into four sectors and the closest and second closest aim point per sector are determined. In a sort of reverse triangulation process, the “tightest” triangle of aim points around a hand position is identified. Relative to the hand’s position, the barycentric weights of the corresponding triangle corner points are then cached. Next, these barycentric weights are used to project the hand’s two-dimensional position back onto the three-dimensional version of the triangle surrounding it (see green and blue triangles). Hence, when actively controlling the X-Z position of a hand, the system automatically interpolates its height relative to the currently tightest surrounding three aim points. Or in other words, when moving the hands, they automatically slide along a triangulated “mountain range”, whose peak and valley constellation is defined by the different aim points. Like this, when moving a hand from one grabbable object to another, it automatically interpolates its height, like one would expect from a real hand reaching between objects. To further increase the usability of the system and to avoid unintendedly “overshooting” the very precise grab points, an additional close-range zone check is performed. In turn, when a hand is inside the close-range zone of a grab point (see green and blue discs around grab points), it will automatically translate to the optimal height for grabbing the respective object. When an object is then grabbed, it will be temporarily excluded from the list of grabbable objects and the whole aiming process. Exceptions to the latter are objects like the ceiling light in Gooze, which have an additional grab constraint space applied. This three- dimensionally defined space controls when a grabbed object needs to be automatically released and additionally it is used to guide the hands along its delimiting edges. Finally, a constant global height interpolation rather unnoticeably interlaces the different sub-systems of the overall aiming system.

This seemingly complicated approach was not conceived at the beginning of working on the issue but was established over developing various simpler but rather unusable implementations. In the end though, this more complex approach led to a rather naturally feeling and unobtrusive interface for three-dimensionally controlling hands with the two-dimensional sub-controls of a gamepad.