A couple of years ago, I made a silly prototype that let people shoot virtual foam darts at their friends’ faces in augmented reality. Although it was a small and fun project, it was the start of my transition from designing 2D UI products for advertisers to being one of the first handful of product designers helping shape what is now the Spark AR platform. It was exciting to see such a simple experience spark joy in people when they used it. Working at Facebook, I can bring these types of experiences to scale on a platform that enables creators to build and share similar augmented reality experiences with their friends and followers. Two years later, I’m still designing for augmented reality and virtual reality — AR/VR — at Facebook, but now I’m working on Oculus products and learning how to design for all of the ways our brains perceive the world.

This transition wasn’t unique to me, and I see it as an industry trend. Based on the number of people reaching out to me recently, it seems more designers than ever are entering the AR/VR space as people realize how transformational this technology is becoming. Let’s take a peek at some key concepts, the general process AR/VR designers at Facebook use and how you can apply it to your own work, as well as how to choose the right tools to use, platforms to build for and how to mind the skill gap to avoid frustration when taking on this new challenge.

On top of that, we construct the rest of our object by adding textures, materials and shaders. This is one of the key differences many designers struggle with when learning a 3D design tool. Unlike with 2D design tools, we’re not applying an image against a flat screen anymore. It’s a texture, applied to a material, tied to a UV map, rendered by a shader. That sentence probably didn’t make much sense, so let’s break it down with imagery.

Say we want to model the “angry reaction” in 3D. We start with a simple sphere model, then unwrap the sphere mesh to create a UV map. Notice how all the edges of the mesh line up to a part of the UV map on the image so it can be realigned later:

Next, we take our 2D image of an angry reaction and apply this to a material on a shader. We then apply this material to a sphere mesh. As you can see, the texture wraps around the sphere nicely.

When it comes to 3D, shaders are probably the hardest component to wrap your head around but one of the most fundamental. Shaders are the instructions given to your device to tell it how to render an image. This is based on all the inputs we mentioned earlier: materials, mesh, vertices, color and light, among others. This happens in every frame to create an animation.

The easiest way to think about this is to think about your favorite 3D video games. You’ve probably seen a game styled more like a cartoon, such as The Legend of Zelda: The Wind Waker, and one styled more realistically, such as The Elder Scrolls V: Skyrim. These styles were determined by the shaders used.

Just like in the real world, lighting defines the brightness, shadows and other properties of an object and surfaces. Lighting is very important for AR/VR as it creates grounding, believability and also helps guide users.

There’s a lot more to 3D, such as rigging, animating and the use of different material types, but this should be enough to help you grasp the basics before diving into a 3D tool.

The Tale of Two Spaces

In a 2D app, everything is tied to the screen position. But in AR/VR, there are (mostly) two spaces. The first is screen space, where an object is tied to the screen, like in 2D apps. The second is world space — it’s an object sitting on your desk or placed in your hand. The concept is simple, but the implications are significant.

Let’s look at typography as an example. A 12-pixel font in screen space is generally 12 pixels all the time, but if we wanted to put text in world space, it changes size and readability drastically, based on how close the user is to it.

What is AR/VR actually?

Let’s break down what virtual reality and augmented reality actually are. Although they are quite similar in underlying technology and exist on a spectrum of immersive technology, let’s simplify and discuss them separately so we can understand the constraints of the systems we’re working in.

AR is about recognizing and understanding the world as seen by the device’s camera. It superimposes media onto the user’s view, combining the real world and a computer-generated one.

Because the system only understands the pixels seen by the camera, it doesn’t interpret the world like people do. Occlusion is an example of an AR constraint. It means the device doesn’t automatically interpret the depth of the world.

While AR superimposes a new world onto ours, VR transports us into a digital one. It does this through a stereoscopic display and headset tracking to make your head into a virtual camera for a digitally rendered world.

The biggest constraint in VR comes from the fact that we’re tricking our eyes and brain into thinking we’re in a virtual world. We need the rules of this world to match our concept of reality.

To simplify, when there’s a disconnect between what our body is feeling versus what we’re seeing, user comfort can be impacted. For example, if you make someone fall in VR when their body knows they’re standing up, this can result in reduced comfort due to the disconnect. Here are examples of how to allow movement while maintaining user comfort.

A design consideration you’ve probably thought about for mobile but that’s exaggerated in VR is designing for the human body. Spatial interfaces use your head and hands to allow you to interact with the world, which is a magical experience and intuitive if done right. However, our bodies have limitations. Looking down, turning around, keeping our arms up — these become tiring over time.

There are numerous domain-specific AR/VR languages and concepts that are best learned while experimenting with the many tools on the market. For example if you want to tackle mobile AR, Spark AR will cover many capabilities and best practices, Oculus outlines concepts specific to VR and whatever video tool you are using will likely highlight how to do compositing to put objects in your real-world footage.

While the language of AR/VR is evolving, this outlines the basics. Now, let’s dive into what it takes to do the work.