Arm Announces Mali D77 Display Processor: Facilitating AR & VR
Arm Announces Mali D77 Display Processor: Facilitating AR & VR
Introducing ASW 2.0: Better Accuracy, Lower Latency
Introducing ASW 2.0: Better Accuracy, Lower Latency
Facebook's Custom Oculus Link Cable for the Quest Will Cost $79
Facebook's Custom Oculus Link Cable for the Quest Will Cost $79
Qualcomm's new wireless VR headsets work with PCs, too
Qualcomm's new wireless VR headsets work with PCs, too
Editorial: Why Oculus Quest Is Exactly What VR Needs
Editorial: Why Oculus Quest Is Exactly What VR Needs

How does Oculus Link Work? The Architecture, Pipeline and AADT Explained

Connecting a USB cable between a compatible PC and Oculus Quest headset enables Oculus Link, unlocking PC VR content for Oculus Quest owners without additional development costs from developers.Oculus Link is a unique intersection between PC and mobile Runtimes. In this article we will share the challenges and solutions designed to bring these VR Runtimes together to offer a high quality, low latency VR experience over a USB cable. We hope you find this article both insightful and interesting.

Building a hybrid architecture

Getting the lowest possible latency in VR requires an end-to-end understanding of everything that’s going to happen in the system -- if latency in the pipeline causes the virtual world to lag behind the player, the result is a bad experience. If, however, latency can be anticipated, the VR experience doesn’t have to lag at all -- instead, the VR Runtime can compensate, mitigating specific artifacts that make such small amounts of latency so obvious to our senses. At the heart of many VR optimizations is a very specific understanding of how every piece of the system is going to work, from the operating system’s scheduler, to the display’s illumination pattern and the mechanics of human perception. To get the maximum performance out of streaming, and to leverage the optimizations that are built into the PC and Oculus Quest Runtimes, Oculus Link needed to have a fully coherent VR pipeline, beginning with the application on the PC and ending with display illumination on Oculus Quest.

Oculus Link pipeline explained

At the highest level, Oculus Link works by streaming inputs and outputs between the Oculus PC Runtime and a client application running on the Oculus Quest.There are no changes to the PC app, or to the compositor running on Oculus Quest. Rather than rendering a backbuffer for the headset display, the PC compositor encodes the eye textures -- surfaces that an application submits to the compositor -- using H.264 video compression, and sends the images as slices to the remote client over USB.

With the use of Oculus, people are able to travel via virtual reality without actually having to pack, fly, and spend the money on a real trip.

On the Oculus Quest, the remote streaming client is responsible for decoding, rectifying and submitting the frame to the compositor, acting as a proxy for the PC application on the remote HMD. The client also sends updates to the PC Runtime with the current HMD position, input state and VSync timings. These are sent without prediction - the PC VR Runtime extrapolates these values to match the target display period.

Effectively, the VR compositor’s responsibilities are split between the mobile VR and PC VR Runtimes, turning the ordinary VR pipeline into a hybrid. The split has two benefits:

  1. Oculus Link is able to leverage the VR Runtime on both sides — rather than incur a blanket penalty for transmitting inputs and outputs over USB, the pipeline has correction at the end of submission that will mitigate some of the transmission latency.
  2. If done right, existing Rift and Rift S applications continue to operate without any knowledge of the streaming backend.

Prediction accuracy challenges

An important responsibility of the VR Runtime is to accurately predict the pipeline’s latency so that the application renders its frames for the target display period. Another important responsibility is to make sure that the frame is never drawn for the wrong display period — if it is shown too early or too late, the virtual world judders and lags in the eyes of the player.

From this perspective, a hybrid pipeline doesn’t work out-of-box — when the two Runtimes are used together, neither is aware of the additional pipeline stages that were added by streaming.

Our solution is a new frame paradigm in which the PC and mobile Runtimes map to one another: from the perspective of the PC Runtime, the streaming latency of USB and the unknown path on Oculus Quest become longer compositor stage; and from the perspective of the mobile Runtime, the unknown path from app render to frame submission looks like a dynamic application.

Frame timing overview: PC

The predicted display time is an important way the Oculus PC Runtime gets the lowest possible latency out of applications. If applications could not assume frames were being prepared for a particular display period, they would need to buffer several frames ahead to account for the unknown pipeline stages that occur after application frame submission.

Unlike on the Oculus Quest, PC VR applications cannot assume a fixed hardware configuration — the run length of the pipeline will always vary depending on what CPU and GPU the user has in their PC. Therefore, the Runtime will create its estimate based on the historical averages of four pipeline stages: the application’s frame loop (CPU and GPU) compositor time, and scanning out to the display.

Virtual reality has captured to the interest of Baby Boomers, Generation Z and Millennials.

For Oculus Link, the Rift PC pipeline estimate falls short. However, the idea can be extended to account for the additional stages that were added by streaming. When a frame is displayed on the Oculus Quest, the streaming client sends the end timestamp back to the PC Runtime; the streaming-pipeline-end timestamp is then substituted for the compositor-end timestamp in the VSync calculation; because the PC Runtime is already equipped to handle variable compositor times, it dynamically adjusts its prediction to match the end-to-end-latency of streaming without having to reduce framerate.

Frame timing overview: Oculus Quest

Normally, the Oculus Quest application receives a pose, renders to that pose, then submits for the target display period. Oculus Link is different in that the predicted frames arrive from a remote device. When a frame’s predicted display time does not align with its actual display time, the result is a misprediction. Mispredictions by a single frame or two are often not visible; however, oscillation between predictions creates judder.

To fix this, Oculus Link leverages the PC Runtime’s Asynchronous Timewarp (ATW) and Adaptive Compositor Kickoff (ACK). ATW guarantees that a frame is always ready for the upcoming display period. ACK makes sure that ATW kicks off with exactly the amount of time needed to hit its deadline. See Volga’s OC5 presentation below for a more in-depth overview of ACK:

For streaming, ACK can be used to make sure that a frame targeting the correct display period is ready to be submitted for every frame interval on the Oculus Quest. When streaming latency is added to ACK’s total estimate of the display pipeline, ATW makes sure to kick off a frame just in time so that it can be decoded, rectified and presented for the upcoming display period.

Delivering higher visual quality with AADT

Delivering high resolution images from PC to Oculus Quest is challenging because the video codecs’ (encoder and decoder) overall latency is directly related to the amount of bits that need to be coded. We could potentially reduce the image resolution to decrease latency, but the lower resolution image will impair visual quality and hinder the experience. We can do better by utilizing the HMD’s lens distortion, along with our understanding of how the human visual system works.

The First Attempt at a VR Experience – The Sensorama. In the 1950s, a cinematographer by the name of Morton Heilig came up with a unique concept he later developed, known as the Sensorama. Featuring an arcade-style theater cabinet, the sensorama was aimed at stimulating a person’s senses. It featured a stereoscopic 3D display, fans and smell generators, stereo speakers, as well as a vibrating chair. The idea of the Sensorama was to fully immerse a person into a film-like experience. Heilig also went on to create as much as six short movies for his device.

Every Oculus headset that has shipped to date applies distortion correction at the final stage of rendering, before displaying the image. This is to counteract the optical-pincushion distortion caused by the lens. As a result, barrel distortion “squeezes” down the VR app’s pixels rendered closer to the HMD display border. This means, for each half of the display, the pixels-per-degree is lower on the periphery than the center of the display.


The figure above shows a sample radial-barrel distortion applied for final viewing on an HMD display.

The final pixel density is tied to the distortion curvature applied by the GPU. For a given pair of lenses, applying this correction is essential for an accurate VR experience. However, for Oculus Link, the image generated by the PC GPU does not have to reflect the exact distortion curvature required by the Oculus Quest HMD lenses because, unlike the Oculus Rift, the Oculus Quest has its own GPU that allows further processing of incoming images. As a result, the Oculus PC Runtime has room to optimize images before transmission. One major optimization comes from the fact that the lenses slightly blur the peripheral regions, decreasing the perceptual resolution. This is where Axis-Aligned Distortion Transmission (AADT) comes into play. Instead of directly sending undistorted (i.e. rectilinear) images rendered by the VR app, the Oculus PC Runtime will use axis-aligned distortion to apply a form of fixed-foveated compression. While radial distortion used for lens correction applies a distortion function based on the distance from the center of the image regardless of 2D direction, AADT applies a distortion function to each axis independently. So AADT makes use of all available real-estate in the rectangular image while radial distortion can show black pull-in on the corners leading to wasted encode-decode resources. Once the AADT image is received by the Oculus Quest, it will undistort (i.e. rectify) the image back to its original shape and size before passing it into the Oculus Quest compositor.
With AADT, a pair of undistorted eye-buffers rendered at 3616x2000 can be compressed down to 2016x1120 saving close to 70% of the video codec budget. AADT not only saves codec bandwidth, but it also saves latency by making sure the encoder and decoder work on smaller images. While these numbers are tuned for the Oculus recommended-specification hardware, they might be updated for better visual fidelity in future releases.

The human visual system is particularly good at picking up flickering in the periphery of the FOV. If Oculus Link directly sent undistorted images to the Oculus Quest, undersampling (or sparse sampling) on the Oculus Quest GPU would cause the periphery to be aliased. With AADT, while the Oculus PC compositor is preparing the image to be transmitted, it makes use of anisotropic filtering with mipmaps. This eliminates undersampling issues and reduces the peripheral artifacts after rectification on the Oculus Quest.

Its not just about the fun and games. Other than providing action-packed, fun-filled entertainment for the entire family, VR plays a big role to help humanity as developers use it to help those in need in ways that were unimaginable 20 years ago. Some use it to cure phantom pain among amputee victims, while others rely on VR to provide therapy for soldiers suffering from post-traumatic stress disorder. VR has also been proven to help children with autism by teaching them social cues and real world lessons. It is being used as a practice board for surgical students before they make the first cut. There is even a VR content targeted at young adults that simulates their physical condition when they get to their 60s, and thus encourages them to save for the future.

Let’s see AADT in action. Observe the grid overlaid on top of the first example followed by a couple of sample screenshots from Oculus Dreamdeck application. After AADT, the undistorted central region is magnified. After rectification on the Oculus Quest, the periphery is slightly blurred while the quality of the central region is preserved.

After AADT vs After rectification/undistortion

Conclusion

In this article we described engineering solutions that enable Oculus Link to achieve low latency, accurate timing and high-quality visuals. We saw how PC and Quest frame composition frameworks are set up to work together for better frame estimates. We also surveyed how headset lenses and human vision work, showing how we were able to optimize bandwidth, while maintaining visual fidelity.

As a reminder, you can always monitor the active PC VR app’s performance statistics including app and PC compositor latencies using the Oculus Performance HUD. This tool will work out-of-the-box for Oculus Quest when Oculus Link is enabled. Please refer to the Oculus Link OC6 presentation below for more technical information and the Oculus Link Compatibility Page to learn more about how to use Oculus Link.

We hope Oculus Link is a great experience for you and your consumer audience, and we look forward to building even more innovative, impactful technologies in the future.

- The Oculus Link Team
What Can We Expect from PSVR 2?
What Can We Expect from PSVR 2?
Oculus Launches ASW 2.0 with Positional Timewarp to Reduce Latency, Improve Performance
Oculus Launches ASW 2.0 with Positional Timewarp to Reduce Latency, Improve Performance
Oculus Quest @ OC6: Introducing Hand Tracking, Oculus Link, Passthrough+ on Quest, and More
Oculus Quest @ OC6: Introducing Hand Tracking, Oculus Link, Passthrough+ on Quest, and More
AT&T to Demonstrate Cloud-rendered SteamVR Content Streamed Over 5G
AT&T to Demonstrate Cloud-rendered SteamVR Content Streamed Over 5G