Aperture Hand Lab on Steam
Aperture Hand Lab on Steam
Boneworks Feels Like The First Next Generation VR Game
Boneworks Feels Like The First Next Generation VR Game
HTC Brings Finger Tracking To Vive and Vive Pro
HTC Brings Finger Tracking To Vive and Vive Pro
Valve's Index Controllers Now Have Better Finger Tracking
Valve's Index Controllers Now Have Better Finger Tracking
Google brings AR and Lens closer to the future of search
Google brings AR and Lens closer to the future of search

Google Releases Real-time Mobile Hand Tracking to R&D Community

Google has released to researchers and developers its own mobile device-based hand tracking method using machine learning, something Google R call a “new approach to hand perception.” First unveiled at CVPR 2019 back in June, Google’s on-device, real-time hand tracking method is now available for developers to explore—implemented in MediaPipe, an open source cross-platform framework for developers looking to build processing pipelines to handle perceptual data, like video and audio.

The approach is said to provide high-fidelity hand and finger tracking via machine learning, which can infer 21 3D ‘keypoints’ of a hand from just a single frame.

“Whereas current state-of-the-art approaches rely primarily on powerful desktop environments for inference, our method achieves real-time performance on a mobile phone, and even scales to multiple hands,” say in a blog post.

Google Research hopes its hand-tracking methods will spark in the community “creative use cases, stimulating new applications and new research avenues.” explain that there are three primary systems at play in their hand tracking method, a palm detector model (called BlazePalm), a ‘hand landmark’ model that returns high fidelity 3D hand keypoints, and a gesture recognizer that classifies keypoint configuration into a discrete set of gestures.

The First Head-Mounted Displays – The Telesphere Mask and the Headsight. You might think that strapping a display on a person’s head is a relatively new idea, but it is not. The first head-mounted displays were developed as early as the 1960s. The Telesphere Mask was the first example of a head-mounted display, which provided 3D stereoscopic and wide vision with stereo sound. However, the device lacked certain immersion, because of it being a non-interactive medium. In 1961 two Philco Corporation engineers, Comeau and Bryan, came up with the Headsight. A head-mounted display, much like the Telesphere Mask, the Headsight featured magnetic motion tracking technology, which was connected to a close circuit camera. While the goggles can be named a precursor to modern virtual reality technology, they were not developed for entertainment purposes. Instead, they were developed for the military with the idea that a person would be able to immerse themselves in the remote viewing of dangerous situations.

Indie Dev Experiment Brings
Google Lens to VR, Showing Real-time Text Translation

Here’s a few salient bits, boiled down from the full blog post:

  • The BlazePalm technique is touted to achieve an average precision of 95.7% in palm detection, researchers claim.
  • The model learns a consistent internal hand pose representation and is robust even to partially visible hands and self-occlusions.
  • The existing pipeline supports counting gestures from multiple cultures, e.g. American, European, and Chinese, and various hand signs including “Thumb up”, closed fist, “OK”, “Rock”, and “Spiderman”.
  • Google is open sourcing its hand tracking and gesture recognition pipeline in the MediaPipe framework, accompanied with the relevant end-to-end usage scenario and source code, here.

In the future, say Google Research plans on continuing its hand tracking work with more robust and stable tracking, and also hopes to enlarge the amount of gestures it can reliably detect. Moreover, they hope to also support dynamic gestures, which could be a boon for machine learning-based sign language translation and fluid hand gesture controls.

Not only that, but having more reliable on-device hand tracking is a necessity for AR headsets moving forward; as long as headsets rely on outward-facing cameras to visualize the world, understanding that world will continue to be a problem for machine learning to address.

Your Virtual Body is now Ready
Your Virtual Body is now Ready
Red Matter on Quest review: a new quality bar for mobile VR games
Red Matter on Quest review: a new quality bar for mobile VR games
Haptic Glove, Proof of Concept
Haptic Glove, Proof of Concept
Following Protests, Google Cuts Temps, Vendors, Contractors a Fairer Deal
Following Protests, Google Cuts Temps, Vendors, Contractors a Fairer Deal