One of the dirty little secrets of designing in-headset virtual reality (VR) experiences today is all the corner-cutting required for the sake of efficiency. Designers want to give users immersive experiences. But the headsets are designed to minimize heat and weight and to conserve energy if they’re battery powered. That means that although what you see through a VR headset feels immersive in a sense (it appears to have depth and to surround you as you turn your head), few things move. And the things that do move are visually simplistic. That corner-cutting compromise is required because adjusting the video stream to show the right perspective on the scene based on your changing pose (position and orientation) is computationally cheaper if there aren’t also objects moving in the scene.
For example, at a recent two-hour forum I attended in AltspaceVR, the content of what the panelists and participants said was compelling, but the simplistic landscape and avatars felt distractingly like a cartoonish game rather than the serious business gathering that the event was intended to be:
And in the VR Fitness app Supernatural, you can work out surrounded by a stunning 360-degree view in the mountains next to an alpine lake — but a careful look reveals that the landscape is actually a still capture (not a video) made more convincing thanks to algorithmically generated subtle water motion overlaid on the part of the image that shows the lake’s surface anytime you happen to turn toward it:
Otherwise, pretty much the only moving objects are a few flying spherical targets for you to strike at as they fly toward you. So while the experience is immersive enough to be wow-inducing, it still feels static compared to physical reality — all for the sake of reducing how much computation the headset must do.
Is there hope for improvement, for extended reality (XR) experience designers chafing at these constraints? Yes.
One common way to make the experience more vivid is to offload most of that computation to a separate device like a phone in the user’s pocket, a laptop nearby, or, for the highest quality, a server in the cloud. And if the experience involves thousands of geographically distributed users having a shared experience, as metaverse advocates envision, servers in the cloud are a requirement for true scalability, not just an option. But that creates a new challenge — how to stream a fire hose of content from the server to the headset.
Last Friday, I tried out a possible solution to the problem: Varjo Reality Cloud (VRC), which becomes commercially available today. With VRC, the headset sends user movement data to a server that knows what’s in the scene. The server can create new frames in response, encode them as a compressed video stream, encrypt them, and send them to the user’s headset. The headset then simply has to slightly crop and warp the frames to compensate for any user movement during the milliseconds it took to hear back from the server, and then display them as a video stream — a much lighter lift.
Until now, Varjo has built its reputation on making the most advanced commercially available VR/XR headsets, delivering experiences sharp enough to match human eye resolution for customers like Boeing, Rivian, Siemens, and Volvo. The company says VRC can now deliver that visual quality even when streaming at a mere 35 Mbps (less than a third of average current internet speed in US households). And Varjo intends to make VRC available to other vendors’ devices too in the future, including low-end headsets and smartphones.
For XR experience designers, VRC opens up the possibility of liberating them from today’s corner-cutting so they can create experiences for users of virtual, mixed, and augmented reality that are visually on par with reality — a step forward that would likely increase the physical appeal of XR and, more broadly, of the primordial metaverse as it emerges.
However, the road ahead is not without obstacles. At one point during the mixed reality demo I experienced while wearing a Varjo XR-3, the stunningly photorealistic full-scale virtual car that had appeared in front of me in 3D and whose gullwing door had gracefully opened began to jitter slightly relative to the floor as I walked around the vehicle. In contrast, when I’d experienced a Varjo demo at an industry event last year, the car was perfectly stable. The difference was that at that event, the stream was coming from a laptop a few feet away, whereas this time it was traveling to my headset from an Amazon Web Services (AWS) server thousands of miles away.
Varjo’s CTO, Urho Konttori, apologetically attributed the jitter to a likely intermittent slowdown in the cross-continent connection that caused the stream to not quite keep up with my physical movements. However, the demo also included a life-size virtual person based on Unreal Engine’s MetaHuman Creator that was so realistic it was light years beyond the silly-looking avatars in most XR experiences — with no jitter at all. And a few days later, Konttori assured me the vehicle jitter problem had been fixed.
The temporary glitch caused me to mentally compare this XR experience based on VRC with the experience of collaboratively editing documents in the cloud. It’s computationally much simpler than XR streaming — and yet, in Microsoft Office, collaborative editing is famously laggy, with any one contributor’s edits often taking multiple seconds to appear on colleagues’ screens. In contrast, there’s zero lag when doing the same thing in competing platforms like Google Docs. For collaborative editing, many enterprises remain loyal to Microsoft despite the frustrating, productivity-draining lag because Microsoft is the incumbent — but in high-quality XR streaming, Varjo does not have that incumbent advantage. On the other hand, VRC has no real competitors yet, and it now has a significant head start.
For experience designers whose medium is XR (and perhaps eventually the metaverse), cloud-based XR streaming could liberate them from today’s corner-cutting constraints, offering a much richer palette from which to create experiences for customers, employees, partners, and other users , like the photorealistic car in the Varjo demo:
One of the dark clouds that today hangs over the potential for a primordial metaverse is the uncertainty about whether mainstream users will tolerate cartoonish experiences, at least for anything other than simple gaming or novelty social experiences. Varjo Reality Cloud just might lift that dark cloud.
Are you involved in designing XR experiences? If so and you’re willing to speak with me about both the challenges and opportunities you’ve encountered, as an input to my research, message me and let’s set up a time to talk.
- Author’s screenshot from the AltspaceVR app.
- Author’s screenshot from the Supernatural app.
- Varjo press kit.