Designing the future: How we prototype in AR and VR

By Rebecca C., Barrett F., Sagar R.
7 min to read
January 19, 2023
A software interface is shown in three different compositions, representing the stages of a prototype. Three arrows point in various directions as a hand reaches in to adjust the prototype.


Get a behind-the-scenes look at how Meta designers are developing new skills to craft immersive environments in the metaverse.

Meta’s company vision is to bring the metaverse to life. A crucial part of that vision is developing new products for emerging platforms like augmented reality (AR) and virtual reality (VR). While designing for these technologies may seem like a whole new realm, the design process is essentially the same as it is for any other experience like websites, apps or games. We begin by understanding the people who use our products, developing solutions and prototyping to test them out with people. After all, whether we’re designing for mobile apps or the metaverse, people’s fundamental needs are the same—they need to understand what they’re using, accomplish their goals and have choices along the way.

Where 3D design diverges from our familiar process is the need to consider how people will interact with spatial experiences in AR and VR. Meta’s design teams have best practices and processes for prototyping 2D designs, but AR and VR don’t yet have established heuristics or interaction patterns, and tools for prototyping in 3D are still nascent. It can take time to become proficient with 3D programs like Unity and Blender, and there is more work to be done than there are designers with spatial design experience. That’s why Meta has invested in developing tools, training and processes to help designers quickly learn 3D prototyping, so they can see what their work looks like in the medium they’re designing for.

3D prototyping also allows designers to iterate and get feedback on their visions earlier in the process, which is especially crucial for VR and other emerging platforms. More often than not, a designer’s real job isn’t just creating things—it’s ensuring stakeholders are on board with the vision that’s presented. Putting a 3D prototype in the hands of stakeholders can be an effective lever to gain alignment and resolve disagreements because a demonstration conveys much more than a static mockup or a document. As Jon Lax, VP of Product Design for Reality Labs, often says, “Demos win arguments.” By investing in a curriculum that allows designers to quickly build and test in 3D, they’re not only learning new skills—they’re empowered to lead in a meaningful and impactful way.

Considering spatial design

3D prototyping helps us feel what AR and VR experiences will be like for people—how they look, sound and behave in Meta Quest devices. There are many elements that are unique to 3D that a 2D mockup just can’t show. For example, is an object in AR or VR within touching distance? How would someone intuitively interact with it? Does it look like it has weight and depth? These are some considerations designers need to keep in mind when designing for immersive, spatial experiences.

Designers that work in immersive experiences need to understand how someone will experience:

  • Proxemics, or how far away an object is from you.

  • Embodied interactions, or how an object reacts when your hand or controller performs a hover or press action.

  • Rendering, or how an object is drawn, textured, and reacts to light.

Our 3D prototyping program guides designers through the process of turning concepts and interactions that are unique to spatial design into tangible virtual artifacts. The toolkits and curricula help designers translate existing design skills so they can gain confidence building AR and VR experiences. By removing the math and guesswork of having to learn 3D tools like Unity, designers across the company can contribute to realizing Meta’s vision of bringing the metaverse to life.

Three animations side by side. On the left, an object moving back and forth. In the center, an object that changes its appearance and scale when hovered and pressed. On the right, an object that toggles back and forth between two different appearances.

Case study: A 3D redesign

Here’s a look at what Meta’s 3D prototyping curriculum entails, and how one designer used it to prototype a fully immersive VR experience for the first time.

Grace J. is a product designer who supports Explore, the first thing people see when they put on a Meta Quest headset. She went from having zero 3D experience to building a fully interactive VR prototype in Unity in just a few months—a real life example of how the 3D prototyping program improves our ability to design in VR.

The role of Explore is to show the breadth and depth of the available apps and experiences on the Meta Quest platform. Grace was responsible for imagining a new product direction that would make Explore feel fresh, relevant and highlight all the things people can do in VR. She knew she wanted to test different aspect ratios and sizes for her designs, as well as test responsive layouts and interactions.

Rather than relying on 2D mockups as her team had done for previous versions, Grace wanted to build an example of what people would experience in VR. Around the same time, the 3D prototyping team announced a mentorship program that would pair an experienced technical mentor with a mentee for one-on-one tutoring. Through that program, Grace was able to build a working prototype to see what her designs would look like from beginning to end in VR.

Here’s the process Grace followed:

  • First, she built designs in 2D using Figma.

  • Then, she exported her work from Figma into Unity using an internal tool that automatically renders designs at the right resolution, and translates button states when you hover and press in AR and VR.

  • To import, arrange and add interactivity to her designs, Grace used prototyping tools like layout regions to ensure that her designs were arranged correctly, and custom components to register hover and pressed states from inputs like hands and controllers.

Two animations side by side. On the left, a panel is shown with different pieces of the panel arranging themselves over time. On the right, a panel toggles between a solid and skeletal version of itself to demonstrate where colliders are placed on the panel.
  • Other prototyping tools allowed Grace to add common functionality to make her prototype feel even more responsive: Scroll regions that let people discover content in different rows and pages, and spatial drag and drop functionality that let people customize their interfaces.

Two animations side by side. On the left, a panel is shown with one section scrolling to the left. On the right, a panel is shown with one piece being dragged from one point and dropped on another.
  • Finally, Grace exported her immersive prototype to a Meta Quest headset and shared her vision with the team.

When her prototype was ready, Grace collaborated with the user experience (UX) research team to show the new Explore designs to people in research sessions using Meta Quest headsets. Showing participants 3D prototypes meant that they noticed details they might have overlooked in a flat mockup, like icons and white space—elements that people normally glance over on web or mobile, but become more prominent in a virtual environment. One participant said that they felt “directed toward the content instead of content being everywhere and me feeling overwhelmed by it,” which ladders up to Explore’s goal as a product. Building a 3D prototype helped Grace deliver new UX insights and provide even stronger guidance on product direction than she could have given using 2D designs. The prototyping process also helped her gain confidence: When asked about the experience, she said, “The ability to prototype in VR is incredibly empowering. It greatly accelerated my ability to quickly iterate on ideas (even 2D ideas).”

Grace’s story is just one of many examples of how prototyping helps us not only design for the unique spatial elements of the metaverse, like proxemics and embodied interactions, but also get valuable insight from real people. Building world-class products for AR and VR means we need to roll up our sleeves and get into the medium directly for our designs to accurately reflect the future we’re building.

Learning AR and VR: Start somewhere

With Meta’s 3D prototyping program, we wanted to ensure that anyone in any design function—be it product design, UX research, or content design—could learn the fundamentals of 3D tools and create their own prototypes faster and more easily than ever before. The products we build benefit from having diverse backgrounds and perspectives, just like the people we’re designing for.

If you’re interested in learning more about designing for AR and VR, there are many resources available to help you get started. For VR, we recommend downloading Unity for free either through Unity Personal or the Unity Student plan and exploring the various tutorials and learning paths that Unity offers to help beginners get started. You can also find creators on YouTube who share in-depth Unity tutorials. If you’re interested in AR, Meta offers Meta Spark Studio, a free software built for creating AR experiences. Here you can find AR toolkits and learning tutorials and also become a Meta-certified creator.

Prototyping is a skill like any other, regardless of the medium. Emerging platforms like AR and VR require designers to utilize that skill in new ways and, ultimately, learn by doing. After all, prototyping is learning—it’s the act of applying a growth mindset to a challenge and emerging with information you didn’t have previously. For designers who want to build the future, the best place to start is wherever you are now.

Design at Meta is for everyone who touches user experience and design.

Whether you’re a product designer, writer, creative strategist, researcher, project manager, team leader or all-around systems-thinker, there’s something here for you.

Design at Meta is a window into the unique expertise and perspectives of the multidisciplinary teams who are building the future of digital connection and bringing the world closer together.