Posing Skeletons and Objects
In VR, you often want to interact via virtual hands with virtual objects. Telling the hands how they should hold those objects can be achieved with different strategies. Here is one!
To achieve a precisely controllable animation of the hand skeletons and thus the hands in Gooze, a universal skeleton poses system was developed. This system includes the three components FTSkeletonPoses (see the image above), FTDevHandObjectPoser (see the image below) and FTUniqueObjectID, several further sub-systems, tools and is tightly integrated with other Components like FTInteractiveObject.
Once FTSkeletonPoses is added to a GameObject (e.g. a rigged hand), one can link any other possibly subordinated GameObject into the Skeleton slot. This automatically recursively creates a list of all respective child GameObjects and the skeleton parent object itself (see green highlight) and applies an FTUniqueObjectID component to each of them, in turn assigning a GUID styled unique ID to each skeleton GameObject. Additionally, the --Start pose is created from this initial configuration of the skeleton and added to the list of poses.
A “pose” basically stores the local position and rotation values as well as further properties for all skeleton objects in a list (see yellow highlight). There are three types, i.e. complete, grab and combinable poses. As the name suggests, a complete pose applies corresponding values to the complete skeleton. A grab pose is a special version of the complete pose, which is individually configured for each hand and each grabbable object. A combinable pose applies only a specific subset of local position and/or rotation values to the skeleton and multiple combinable poses can be applied on top of complete or grab poses.
A precisely controllable script-based interpolation system can then animate, in a forward kinematic fashion, between multiple poses and mix them in various degrees. Due to the setup of the skeleton, this in principle “simple” interpolation of local position and rotation values of the skeleton GameObjects results in believable animations. Only the caching of animation states and pre-calculations for mixing poses and precisely controlling and prioritizing them added a certain complexity to the animation system.
In simplified terms, this system is based on a “current” and a “target” pose, which can be freely switched and between which the skeleton can be precisely interpolated. On top, the combinable poses can also be interpolated into these mixed values and possible overlapping pose instructions can be controlled via setting up priorities. In code, timed animations can be triggered in real-time and when a pose is switched the system automatically applies a custom animation from the current possibly mixed pose to the new configuration. The current configuration of poses, interpolants and priorities can also be manually adjusted in the editor (see pink highlight). The right hand in the top image is posed according to the pose configuration shown in the pink highlighted area: The hand is posed in-between the Default and the Fist pose and the combinable PointingIndex pose (affecting only the index finger) is calculated into this mix of values, too. To provide flexible usage in various situations, the animation system can be further configured to animate during Update and/or FixedUpdate steps. Additionally, it can continually animate on every corresponding frame or process animation only, if there are pose changes scheduled (see green highlight), to reduce its impact on performance.
As many poses needed to be configured and stored for the Gooze demo – e.g. two poses per grabbable object, one for each hand – and a pose consisted of a rather huge amount of values, a comprehensive editor workflow was required. This workflow should automize as much of the process as possible.
To create a complete or combinable pose, only the FTSkeletonPoses component is needed. In case of a complete pose, one needs to manually adjust the skeleton objects (the local positions and rotations), so the skinned mesh is posed as needed, enter a pose title and click the save complete pose button (see light blue highlight). The pose will then be added to the lists of selectable poses and can be freely interpolated to and from. To create a combinable pose the record combinable pose button needs to be clicked initially. Once this mode is started, each individual parameter change of the skeleton objects is recorded. When the skeleton is posed as needed and a pose title was entered, a click on the save combinable pose button (see light blue highlight) will internally store a complete pose but set up corresponding flags only for the actually changed parameters (see yellow highlight).
The workflow of creating a grab pose is slightly more elaborate and involves the editor only FTDevHandObjectPoser component, which was developed purely for this purpose. Once FTDevHandObjectPoser is added to an empty GameObject, by clicking the respective button, it can automatically acquire its configuration from an FTGameControllerHandsController component present in the scene (the component which is equipped with the corresponding hand objects). This creates duplicates of the correctly configured hand objects and parents them to the empty GameObject. This is to avoid breaking any prefabs or other configurations. For similar reasons one needs to create two further duplicates of the grabbable object and link them to the corresponding slots (see the image above). In the following, one needs to pose and position the object and the hand for one side, so they seem to be naturally set up. By typing in a correct pose title and clicking on save complete grab pose in the respective FTSkeletonPoses component, the grab pose specific to this grabbable object will be saved in the duplicate hand object. Likewise, clicking on the respective configure object for grab button in FTDevHandObjectPoser will save the grabbable object’s position and rotation relative to the hand in the corresponding section in FTInteractiveObject.
It is possible to configure two completely different grab poses, when grabbing the same object with the left or right hand. E.g. the cup in the Gooze demo is either held at its handle or its body, depending on which hand is grabbing it. Nevertheless, most objects were supposed to be grabbed in similar but mirrored poses. To avoid needing to pose the hands twice, a respective feature set was implemented. Using the corresponding buttons in FTSkeletonPoses one can not only copy and paste poses from one component instance to another, but also automatically mirror these poses alongside three mirror planes (X-Y, Z-Y and X-Z, see light blue highlight in the top image).
So, once both hands and objects are posed, and the respective poses and grab configurations are saved in the duplicate GameObjects, they need to be transferred to the original GameObjects in a convenient way. For that reason and as already mentioned, FTSkeletonPoses offers a feature to copy and paste single or several poses between component instances (see light blue highlight in the top image). Likewise, it is also possible to copy and paste left and right hand grab configurations between FTInteractiveObject instances (see green highlight in the image below).
FTInteractiveObject
The FTInteractiveObject component is the base class for all interactive objects in the Gooze demo. These include e.g. the cup, the polaroids and the ceiling light, but also the invisible hole in the wall etc. The component provides various options to configure the object for the diverse sub-systems of the application, like spatialized audio, thought subtitles, object grabbing and object resettlement (i.e. the sub-system, which automatically places the object back at its initial position, when moved and released). To simplify adjusting the maximum distance from a hand’s grab point to the object’s Collider, inside which it can be grabbed, a grab distance gizmo was implemented (see yellow highlight in the image above). To roughly visualize this distance, the gizmo automatically renders a semi-transparent dynamic duplicate of the mesh, in which all surfaces are outwards scaled by the respective distance (see screenshot of cup with green gizmo).