Introduction
Spatial computing is having a moment! With the announce of the Apple Vision Pro, mixed reality is starting to capture some mainstream attention, and if that's you -- welcome! The promise of spatial computing is that we can move away from flat screens for those situations where it would make more sense to interact naturally -- with hands, voice and gaze. The upshot being we can spend less time staring at our phones and more time, you know, actually being human! Which of course isn't every use case -- I sure as heck don't want to spend hours manipulating Excel spreadsheets by waving my arms around!
<Source: Minority Report>
Where it does make sense to use spatial computing is when we need to interact with 3D content or with our environment. For example, I can have information about all of the plants in my garden show up over the actual plant, with a dial to adjust nutrient levels.
But it can seem overwhelming to jump into it -- even for experienced developers. I'm making it my mission to help everyone who wants to develop spatial applications to be successful. So, let's start from the beginning (which is usually a good place to start), with the question "what even is spatial computing??"
What is Spatial Computing?
You've probably been hearing a bunch of various _R terms being thrown around by now: AR, VR, MR, XR. Let's clarify what we mean with each of them.
VR (Virtual Reality) is straightforward enough -- it is a fully immersive simulation that puts you inside a digital world.
Your physical movements are translated into digital actions and you are shown graphics on a VR headset that updates as you look around. This is functionally like being in a video game. Hardware examples include Quest, Vive and Varjo headsets, and software examples include VRChat, Beat Sabre and Superhot.
AR (Augmented Reality), on the other hand, overlays digital information and virtual objects onto the real world.
Sometimes AR applications provide the ability to interact with this digital content via a touch screen or other input device. Often, AR content will be anchored to a specific point through the use of image tracking -- either a QR code or something else. You may be surprised to know that AR is actually a huge industry already, including examples like Snap / TikTok filters, Pokémon Go-like games, and advertising.
MR (Mixed Reality), also known as Spatial Computing, is a paradigm where you can use physical interactions to manipulate digital objects and vice versa.
This second part is an important difference from AR that we can call world understanding. In AR, we can attach content to an image and can do some basic things like detecting the floor to provide the feeling that the digital content is situated in the real world. MR takes that to the next level by understanding more about the specific space. In the garden example, it's not just that I can have a gnome walking on the ground, but that the gnome can lead me to the plants that need some extra attention. And by having this additional understanding, we can use digital interactions to affect physical changes. Remember our example of increasing fertilizer? We can also turn our garden lights on or off based on the location of our garden gnome. So far examples are limited but include hardware like the HoloLens, Magic Leap, and the upcoming Vision Pro, with software being primarily targeted to industrial use cases, like this one from Toyota:
<Source: Microsoft>
And finally XR -- alternately "Extended Reality" or just x as in the variable, this term is meant to encompass all of the other terms we've mentioned under a single umbrella. Because although these technologies are all different, they share a lot of similarities. Mainly that the main method of interaction is using natural user interface -- head-mount displays, hand tracking, eye tracking and voice.
How does spatial computing help me?
Great question. We talked about a few positive and negative examples above, but how can we decide if spatial computing is the right approach for a particular use case? I propose the following razor:
How much does spatial context matter?
So if we go back to our examples -- how much does spatial context matter for Excel? Basically not at all. This would be a poor choice for spatial computing. How about gardening? In this case, the spatial context matters a lot -- the location of the plant is just as important as the stats about the plant!
How about a few more examples to illustrate this point:
Training in a factory environment? Great use case! It matters a lot that our digital content can be aligned precisely to the factory equipment.
CRM? Less great! Here we want to track customer data and their relationship to us. If the customer has an associated location it is treated more as an abstract concept.
Escape room? Great! Interacting with puzzles at physical locations is what the whole experience is about!
Open World RPG like Skyrim? Less great! This type of experience may be better suited to VR, where we can immerse the player completely in a fantasy world.
Walking tour guide? Great! We can add digital content to points of interest along our route, and even have a virtual character walk the route with you to point things out!
And I could go on! But hopefully that gives you some intuition for how spatial computing can help make certain experiences easier or better.
Okay, so now that we have some understanding of what it is and what it can do for us, what's the problem?
Building spatial computing applications is hard.
Well, at least that's the perception. And it's partly true! So what are the main difficulties that people actually run into? Let's look at the components we'll need for creating a spatial experience:
Design -- what is it you want to achieve? Why would it be better or faster in mixed reality vs. a flat screen?
Digital Asset Creation -- how do we create the digital assets that we will be using in the experience?
Physical Integration -- how do we anchor our digital content to a physical location, and how do we interact with that physical world?
Experience Creation -- how do we put it all together into a coherent experience?
Publishing -- how do we share our creation with the world?
It seems like a lot, but getting started can actually be easy, as there are a number of tools and learning resources out there to help! Most of these requirements are actually shared by video games, so those are the resources that we'll be tapping into. For content creation, we have tools like Photoshop, Blender/Maya and Wwise/FMOD (to name a few). And for tools to put it together, we can use Unity or Unreal Engine for cross-platform native development, or a WYSIWYG editor like Elysium or Enklu for a drag-and-drop experience.
We'll take a deeper dive into the ecosystem of available tools next time and break down how to start creating step-by-step. Until then, I hope I've inspired you to start thinking about how spatial computing can transform your world with the intersection of physical and digital.
Follow my spatial computing journey as I post updates here, watch me hack away on projects on Twitch, or join the newsletter. I hope to see you in mixed reality soon!
Allons-y!