Comments made by Oculus CTO John Carmack at Oculus Connect 5 and Twitter suggest that the upcoming Oculus Rift S cameras could support finger tracking.
To be clear, this doesn’t mean the software to do so would be ready by the time Rift S launches. But it does mean the headset could one day recieve the feature as a software update in future- the cameras are seemingly suitable.
Rift S Onboard Cameras
Rift S was first revealed in a TechCrunch report in October. The report revealed the headset would be an iterative update, increasing resolution and changing to the same inside-out tracking system as the upcoming Oculus Quest.
Earlier this month we confirmed the TechCrunch report by discovering references in the Oculus PC software code to a ‘Rift S’ with onboard cameras.
During 2018’s Oculus Connect 5 conference, Oculus CTO John Carmack briefly spoke about a project to bring finger tracking to Oculus Quest:
This seemed to suggest that the main limitation for finger tracking on the headset was the power & compute limitations of the mobile platform. He stated that Quest can be used as a platform to research finger tracking for future headsets with “the power and ability to do real time”.
Stanley G. Weinbaum is a well-known science fiction writer from 1930’s told us about this technology in his short story named Pygmalion’s Spectacles. His work made him a true visionary in the field of Virtual Reality. The story shares the idea that the wearer of the goggles can experience fictional worlds, even before the official term was coined.
The TechCrunch report and our findings suggest that Rift S will feature the same Insight cameras as Quest. So given the enormous relative power of a PC, shouldn’t Rift S be capable of finger tracking?
To verify we weren’t misunderstanding his comments, we reached out to Carmack on Twitter . Here’s how he responded:
Carmack doubled down on stressing the power limitations of mobile as the main barrier. When asked about the featue in regards to PC, he confirmed it was possible. Requiring a reserved CPU core or two rules out this tech being usable in CPU-intensive apps. But it would be entirely possible for developers of apps suiting finger tracking to optimize for this limitation.
If you want to purchase an Oculus Rift right now on Amazon.com the VR headset may not ship until after the Game Developers Conference in March. “Rift continues to be available across our channels, at this time mainly online (Oculus.com, Best Buy online, Amazon, etc.),” explains an email this week from a Facebook representative.
These comments aren’t indicative of the feature actually being available at launch- or ever. But what they do suggest is that the camera hardware is suitable and PCs are capable.
A Long Researched, Challenging Feature
The first indication of Facebook’s interest in finger tracking for VR was revealed in late 2014. The company acquired startup Nimble VR , composed of four veterans of hand tracking technology. But upon this acquisition, Oculus was clear that this tecnology “may not even be used in the CV2 or CV3”.
That’s likely because Nimble’s tech wasn’t just software, but a dedicated depth camera. Depth cameras deliver excellent finger tracking such as that found in the new HoloLens 2 AR headset. But the high end solutions are costly and the low end ones are bulky and relatively heavy, adding weight at the worst possible area of a headset (directly in front).
Movies and sports and other events will be viewable in VR as technology continues to advance.
Delivering high quality finger tracking on regular cameras is an entirely different level of challenge. However, if it can be done it allows finger tracking to be added at no extra hardware cost to headsets which already use cameras for other tracking.
In 2016 at Oculus Connect 3, Chief Scientist Michael Abrash made a range of predictions about VR in the year 2021. He noted that finger tracking could be done perfectly with markers on gloves (and showed this off), but not yet directly. However, he predicted that by 2021, it would be possible:
We’re now just two years away from 2021. This lines up with our expectation that Rift S could one day receive an update to support it.
Machine Learning Breakthrough
Last year at Facebook’s annual F8 conference the company finaly showed off high quality gloveless, markerless finger tracking:
An Oculus representative explained on stage that this “breakthrough” was made possible by Facebook’s heavy investment in machine learning (“AI”) research. They use the kind of glove system described by Abrash in 2016 to train a convolutional neural network to recognize hand poses.
To your great surprise, the concept of Head Mounted Display is also not a new idea. The first head-mounted display was developed around 1960’s. the Telesphere mask was the first example of a head-mounted display, which provided 3D stereoscopic and wide vision with sound.
The results look impressive. However, Facebook hasn’t demonstrated this technology to journalists or developers yet. We therefore don’t know how well it works outside a lab environment.
Just over five months later at Oculus Connect 5, Carmack made his statements about finger tracking on Quest, seemingly referencing this project.
Valve’s Headset & Vive Cosmos
Facebook isn’t the only PC VR company with finger tracking ambitions of course. In October last year HTC announced finger tracking is coming to the Vive Pro using its two front facing cameras:
HTC will show off this finger tracking at GDC next month. Of course, the Vive Pro‘s four figure price puts it out of reach for most consumers. But the upcoming Vive Cosmos also has cameras on the front, and it may be significantly more affordable. That opens up the possibility that Cosmos could also have finger tracking.
In November last year, images leaked online of a prototype Valve VR headset. Sources close to UploadVR confirmed this was an in development first party headset. Not only did the headset have two front facing cameras, but it also had a slot remarkably similar in shape and configuration to the Leap Motion finger tracking addon.
The Royals Are Also Using It. You know that virtual reality is big when highly prominent people are also getting in on it. In March 2018, Prince Harry and Meghan Markle honored International Women's Day by encouraging young women to study science and technology. In the process, they attended a school and tested out a virtual reality set. The couple had a positive experience with virtual reality. They both appeared to enjoy learning about the technology and how the headset works.
Given that startup VR headset company Pimax offer a $169 Leap Motion addon for their 8K and 5K+ headsets, it could be possible that Valve plans the same. Or perhaps Leap Motion was simply being used to help train a first party algorithm using the front facing cameras. If the Valve headset releases this year it could be direct competition for Rift S, so it will be interesting to see what features it offers.
When people put on a VR headset, one of the first things they do is reach out with their hands. Tracked controllers offer a basic representation of our hands. They’re also very suited for gaming. But they don’t track the vast majority of finger motion and the very act of holding them restricts those motions too.
In non-interactive VR experiences and social VR, controllers often feel more like a chore than a help. If we could enter these experiences by just putting on a headset and seeing our real hands, this reduction of friction would be a welcome improvement.
Enterprise customers using VR for tasks like architecture visualization often forgo controllers entirely due to this friction. Gaming-style controllers are simply much less appealing to non-gamers than directly seeing your hands.
In social VR the ability to gesticulate freely with your real hands adds to immersion and increases social presence. I’ve spent a lot of time in platforms like AltSpaceVR- Leap Motion is by far my preferred method of input.
Facebook purchased Oculus VR for $2 billion in 2014.
Given Facebook’s tone when discussing the research it seems significantly unlikely that Rift S could have finger tracking at launch, or even soon after. But given Carmack’s comments, when the software becomes ready and if Facebook wants to, it could bring finger tracking to Rift S one day.
Tagged with: facebook, finger tracking, hand tracking, HTC Vive Cosmos, john carmack, leap motion, oculus, oculus rift s, valve