How Video Games Inspire Great UX

“Perspective is worth 80 IQ points.”

Alan Kay

This quote is meaningful to me as I’ve often struggled with a problem until an insight from a user test “changed my perspective”.

I had this intuition that video games would also change my perspective. They seemed so much edgier, playing with deeper and more experimental UX techniques. I wanted to go well beyond classic “gamification”. That was all the rage a decade ago where apps used toys like “achievements” to encourage app use. Of course, given what we’ve learned in the last few years about the overuse of mobile phones and social media, it also seems irresponsible to make our apps “sticky” or “addictive”. I was looking for something deeper. Games felt more visceral and I wanted to understand them better and see where it took me.

I’m an avid gamer but they felt a bit foreign to me as a UX designer. For example, one of the first game terms I learned was “juicy”, which refers to the amount of video effect you get for any action. Juicy games feel just a bit overwhelming but in a good way. This might be fun but likely counter productive. I wasn’t looking to overwhelm the user.

My other concern was that games create tension. When you approach a fight in a game, you need to master a range of skills or your character will lose. If you were to ask any competent UX designer to design the perfect flow to beat a monster they would simply have a single button “Beat Monster”.

That’s the paradox I was facing. Games felt like they were about sparkles and tension. Great app UX is about minimalism and simplicity. Fortunately, I found Raph Koster, the author of A Theory of Fun. Raph is known as a “Game Grammarian” and deeply deconstructs how games are made. His book is a very game-like exploration of how games work. Much like how Understanding Comics is a comic about how comics work.

I met with Raph several times over 8 weeks where we took his book apart, applying it to my world of UX design. He was energetic and excited to find a parallel world that could benefit from his work. I’ve been in UX design for over 30 years and I was a bit, well, set in my ways. Raph was patient and in a good-natured way enjoyed skewering my misconceptions. This article is my journey fighting through these misconceptions.

Misunderstanding 1: Just copy cool tricks

The game Horizon: Zero Dawn has an amazing heads-up-display UI. It just seemed obvious to me that there was potential for mobile apps. Raph’s point was that video games create these new UIs all the time. It would be a never ending list if I just copied what games are currently doing. His advice was to go a bit deeper and first understand how games create these types of UI.

Misunderstanding 2: Games are linear and plot driven

Games have the ability to force situations, such as running into a canyon and having nowhere to go but up a ladder. Apps on the other hand, usually have the opposite, offering a broad toolkit of choices. Games, I thought, can exploit narrative to force situations which made their life easier.

Raph said yes, games do use plot (they are a form of entertainment after all) but his point was that great games go so much deeper than just plot, usually working on low level mechanics first. You only have to appreciate the 12 ways to jump in Super Mario Odyssey to appreciate his point.

Misunderstanding 3: Gamers practice over and over. Games have it easy

Raph laughed when I said this. “You realize”, he said patiently, “that most games fail?” This made perfect sense the moment he said it. Of course, only the best games reward practice. You have to design a great game to get people to have the confidence that practicing is worthwhile.

The upshot is that the Raph convinced me to forgo any quick and easy ‘cookbook of tricks’ approach to this problem and go deeper and understand better how games are built, from the bottom up. His book covers a wealth of material and I encourage you to read A Theory of Fun for more detail. But we came up with six “lenses” that cover the transition of his book to UX design.

I want to be clear that video games are not exactly like applications. The goal here isn’t to copy games but instead be inspired by them. Many UX designers devour Edward Tufte’s books on information design not because they expect to visualize complex data sets themselves but more by the joyful intellectual insight that comes from understanding data patterns he describes so well.

Lens 1: Story vs Narrative

The first point Raph made felt a bit overly specific, calling out the difference between the words “Story” and “Narrative”. They felt quite similar to me but his point was that “Story” is author generated while “Narrative” is user generated. Game designers create a series of events (A-B-C) you have to pass through but what users actually do is often a surprising pattern of:

• staying on on step A for a very long time
• skipping step B entirely
• circling around step C multiple times.

At first I thought he was just describing classic user testing. Of course users go through your design in ways you didn’t expect, all experienced UX designers know this. But his point was deeper: it’s not the journey, but how users recall the experience. Users will always create narrative. It’s just human nature, they will go through any experience and remember it by constructing a narrative. This is what game designers understand deeply. If your story has huge gaps and doesn’t lead the user carefully then users can’t help themselves, they will create their own narrative that fills in the gap.

Most apps today will just “throw” 5 new features into an app with little connection between them. Users however, will create a narrative to help them understand what they are seeing, even if none exist. What narrative could they construct? They are likely making up stories that would frighten you. Anyone who has done user testing appreciates this. Users ‘connect the dots’ in ways that will astound, often creating mythologies that were never intended.

APPS
Just throw in a bunch of features into a pot.

GAMES
Understand everything is a journey. Work hard to make everything a closely connected arc of events that help the user create a narrative that matches the overall story.

EXAMPLE 1:

The best example of this is the original 1984 Macintosh boot sequence. Every PC of it’s day had a simple power switch which just ‘beeped’ when turned on, followed by a bunch of streaming and confusing text. The Mac booted into a startup screen showing only a pattern with a smiling Mac. Nothing more, no text, no gibberish. Eventually the desktop appeared but it was just a tiny variation, the same desktop but with an added menu bar, a disk icon and a trash can. The boot sequence started with a promise that resolved into a working desktop.

Not only that, but the corners of the display were rounded which made it look more like a desktop ‘blotter’ which were popular back then.

Even more, the trash can added depth to the illusion by getting ‘stuffed’ when you put something into it. The Mac took a very hardware driven concept, turning on your computer, and turned it into theater. Yes it had the boot sound, but it then showed a promise, a compromise of the final desktop and as it booted, ‘inflated’ that promise with the final working model. Why people loved the Mac is often misunderstood. I’d claim that it’s this dedication to taking people on a carefully crafted story, one which allowed users to craft a compatible narrative, that is at the heart of this devotion.

EXAMPLE 2:

Another more modern example is this landing page for PayPal. Notice how the page clearly invites you to choose. Are you a “Personal” user or a “Business” user? As you mouse over each section, the story unfolds, expanding your choices, offering you things you can easily understand and identify with. Each branch has a clear call to action. This is a beautiful story telling sequence that pulls you in and gets you to become an active part of the on-boarding process.

Lens 2: Games are fractal

Raph uses the game Frogger to explain what he means by “Games are fractal”. This single screen is composed of several levels. At first you are in a safe starter area at the bottom where you can move back and forth freely with no risk. Second is the street, where you have to dodge cars followed by a river which is a bit trickier as both the road and the ‘cars’ are moving. Finally you get into a ‘parking’ area where it’s easy to get into the first one but progressively harder as they fill up.

But he literally means ‘fractal’ in that the game isn’t a simple linear “cross the road, cross, the river, and park your frog sequence. Of course, at the surface level, that’s true but his point is that it’s a nesting cascade of discoveries. 

To win the level you must first cross the street. To cross the street requires that you move the frog. To move the frog requires that you understand joystick timing. Each of these sub levels have their own feedback considerations:

• Street: the cars movement
• Frog: How it moves, how far it jumps each time
• Joystick: Direction and speed of movement (it’s quite slow actually)

Games understand that each of these levels has their own set of feedback, motivation and learning that must take place. This level of deconstruction, in a 30 year old game no less, blew my mind. Games were complex! They really paid attention to detail. There was a lot here to understand.

Compare this to app designs. For very good reasons, we try very hard to fall back on guidelines. We work very hard NOT to reinvent the wheel. While this makes sense, we can’t just fall back on guidelines forever now can we? Maybe this is a bit obvious, but we’re not going to be using today’s phone UI designs 30 years from now.

How do we evolve and create new UX patterns? What’s inspiring about games is that they are trying to evolve all… the… time. While clearly this would be a chaotic with commercial software, I still find it inspiring that games are built on this core belief that they rigorously unpack each action. Everything you do is multi-layered and must be unpacked, improved, and most importantly, taught to the user. While we may not want to emulate the speed of how games change UX patterns, we can certainly learn from their deep commitment and dedication to detail.

APPS
Follow your guidelines, be consistent, don’t rock the boat.

GAMES
Rigorously unpack everything, don’t worry if you sink the boat.

EXAMPLE

The computer example here is desktop menus. “Selecting a menu item” is actually a fractal cascade of skills where you first start horizontally browsing the menu bar, with a click, you shift into a vertical mode but keep the same basic highlight approach. For hierarchical menus, you need to understand the graphic hint that there is something deeper and then navigate over to reveal and then select that menu. Anyone who has taught beginning computer users the menu system knows how hard it is to master hierarchical menus. It’s takes practice to find, reveal and track over to that menu. There is a fractal cascade of skills required.

Lens 3: Learning Loop

The Learning Loop is a basic concept in psychology for acquiring any skill. You start with an intent such as “shoot the laser”, you take an action, such as pushing a button, and you see the result, the canon powering up and shooting a beam of energy. This loop is fundamental to learning as you have a model of how things work, then interact with the affordance in front of you and the resulting feedback then loops back and updates your mental model.

Far from being a dry academic model, this is the foundation of how learning happens in games. Raph has a great quote in his book for this: “Fun is just another word for learning”. In order to have fun, you must learn. I find this inspiring as app design wants your users to learn but we’ve rarely appreciated this could be fun.

Games understand that in order to learn you must start thinking in layers. Begin with a basic skill and slowly add more, getting better one layer at a time. But the key insight is that these layers are built one on top of the other. The classic example is Super Mario brothers. 

You start off locked behind a pipe, in a safe zone much like Frogger. Here you move and jump, getting nowhere until you figure out how to jump on top of the pipe. You’ve just learned the first basic ‘Loop”: jumping. As you move to the right, a low ceiling appears and it’s very easy to jump into the ceiling, which hits one of those “?” bricks and a prize drops. You’ve learned your second variation of the loop: Hitting. Finally, when you encounter your first monster, it’s not a far stretch to assume you can jump onto them, and “Tada!” you just zapped your first monster. Loop 3 is attacking. There are clearly 3 distinct versions of jumping going on here:

• Initial jump. Simple button press
• Long jump. Long button press
• Landing jump. Timed jump

What’s so interesting here is that there is only one ‘thing’ you’re learning: jumping. But by stressing subtle aspects of how to jump, the game builds up variations of it. A basic jump gets you over things, a long jump can “open” and landing a jump can “attack”. A boring app designer like me would assume you’d need 3 different verbs/buttons for this but Super Mario does this with a single “Jump” action.

Each jump has their own triggers but to the user “it’s just jumping”. Crafting a careful learning loop enables people to learn a range of tasks under a single concept.

Nintendo is the master of this, doing it not only for Super Mario Bros but for all of it’s games from Super Mario Odyssey, to Splatoon, to even Luigi’s Mansion (which I doubt any of you have played). Shigeru Miyamoto, the esteemed director of Nintendo games has this amazing quote:

“That’s how we make games at Nintendo: we get the fundamentals solid first, then do as much with that core concept as our time and ambition will allow.
As for the courses and enemies, those actually came at the very end. They were done in a single burst of energy, just thrown together, almost.”

Shigeru Miyamoto(Nintendo Games Director)

This blew my mind when I first read it. What do you mean both the game levels and course were just ‘thrown together’! That’s insane. But it shows how much Nintendo focuses on the core building blocks. It’s what makes games work. I think there is a huge lesson for us here.

APPS
Each feature is in isolation, how it is done usually has little relation to other features (other that using a style guide).

GAMES
Build a game through a single, mechanic that grows in expressive power by adding modifiers like time, special keys, or timing.

EXAMPLE 1:

In a somewhat ironic twist, when the Mac first shipped in 1984, it used games to teach users how to get started! 

The goal was to start simply and teach the first loop: moving the mouse. It then progressed to clicking, menus, dragging, and even shift clicking. What struck me was how deep and varied the loops are for desktop. This rich input model allows for much more sophisticated positioning of the cursor, better text selection, and input control. While we all agree that the Desktop UI can be overly complicated for some users, it’s important to appreciate how rich and detailed it’s loops were. In our discussions, Raph specifically called this out, saying that mobile has a lot to learn from desktop.

EXAMPLE 2:

But mobile does have a few tricks. People often think of “tap” and “pinch” as separate gestures but they really are just variations on the same loop. Here is an example showing both in Google Maps:

Notice how when the user pinch zooms in, they can still drag the map around like they do with single tap dragging? Pinch is just a high order loop on top of drag. 

Lens 4: Affordances

Affordances has a classic definition in UX, popularized by Don Norman in his book the Design of Everyday Things. Games have a subtle spin on this: affordances prompt the things you already know. Let me use a few examples from “Breath of the Wild”, one of my very favorite games. Like many exploration games, you’ll get a clear and obvious prompt when you stand next to something:

This “Take” prompt will occur thousands of times throughout the game. You clearly learn how to pick up an item fairly quickly but this prompt also gives you a clear indicator when you can pick it up as well.

There is a more subtle version of this when fighting. In this case, the robot will always swing wide before it attacks from the side, signaling when to dodge. This is something that you need to learn in order to get good at fighting robots. But once you learn it, it’s a very regular pattern you can count on.

But my favorite example was one of the core innovations in Breath of the Wild: the stamina wheel. It is a simple circular gauge indicator of how much ‘stamina’ you have. When you start climbing, the gauge starts to empty. If you don’t get to the top before the wheel runs out, you fall.

What’s amazing here is how the wheel was implemented. At a simple level it’s “just a gauge” but it’s far more sneaky than that. If you climb, you’ll notice a range of feedback on this affordance:

1. The character leans in when climbing
2. When stopped they lean out and the wheel stops
3. During a steep section, beads of sweat appear and the stamina drains faster. In addition, they lean in and appear to be ‘trying harder’

This basic example of ‘just climbing’ shows how subtle game affordances can be and how clever they are at using varying types of feedback. At a basic level, they really aren’t that complex, showing a simple resource: stamina. However, it’s by having the next level of detail, showing, in effect, the second derivative of stamina usage, the game is teaching you to take the less steep route. That extra bit of feedback, the leaning in and the sweat, helps you understand that you’re using up stamina faster.

I had played this game for over 100 hours, and had clearly learned to take the less steep sections when climbing but here’s the key point: I had no idea I was doing it! The game taught me without me knowing I was learning. This is serious Jedi mind trickery. 

This level of detail is what makes games so intriguing. Raph talks about the many types of affordances in his book, but I’d just like to focus on five types of affordance feedback that apply to app UX:

1. Multi-variant
2. Reality
3. Multiples 
4. Mixed
5. Grace-notes

Multi-variant

Games work hard to give you lots of feedback hidden in the action. From the sand particles flying out when Mario runs, to the cloud trail he leaves behind, to when his arms fly out at certain speeds, Super Mario Odyssey works hard to give you all sorts of variables in subtle ways. This was exactly what happened with the stamina wheel above. By encoding things cleverly, the user can pick up on multiple aspects of the game.

Reality

Games work hard to map the right actions to the correct aspect of the controller, which has a rich selection of triggers, analog and digital joysticks, to simple buttons. Like Desktop, games consoles have a rich set of input loops to use. Mobile in contrast just has tap, drag and pinch. Mobile is impoverished compared to games and even desktop: we are a prisoner of our flat slab of glass. And don’t get me started on long-press, it’s very existence is proof that mobile has an expression problem (making me feel like old-man Simpson).

Multiples 

Games work really hard at using visual, audio and haptic feedback in games. Why don’t we? The obvious answer is that users often don’t like it but we have to ask ourselves, how can games get away with it and we can’t? Yes, games have it a BIT easier as the volume is turned up but it’s worth asking, what is it about games and how they do it that it feels obvious and fun whereas we do it and everyone hates it. It’s arguable that we’re being a bit too heavy handed. We need to be a bit more subtle and nuanced.

Mixed Styles

Games work very hard to mix graphic styles. The good guys in Mario are soft and rounded while the bad guys are sharp and pointy. In the game Horizon Zero Dawn there are two very different graphic styles:

The more ‘cave painting’ style represented the main character, which came from a primitive tribe, and the sharper more technological style was for the items in the environment she could ‘see’ with her heads up display. It works well to give you a “me vs the world’ feel to things on the screen.

How could we use this? Here is simple web page example:

The title bar is clearly visually distinct from the body but stops there. There should ALSO be a visual difference from the left nav and the actual body content. The left nav should be a lighter blue so it ties into the title bar. It would give a clear visual style about the title being most important, the nav next, and finally the content.

Grace-notes

This is the last and most playful example. Games create small light moments to add ambiance to a game but also to drive learning. In Breath of the wild, grasshoppers fly out of the grass when you run through it. This is just atmospheric but later when you learn to ‘cut the grass’ you can actually find those grasshoppers before they fly.

In Half Life, you can flush the toilets. It doesn’t do a damn thing, there is zero value to do it but it does make the world feel more interactive. That way when you walk up to a closed door, you’re more likely to try opening it, I mean the toilets flush, why not try?

This is clearly a very subtle but powerful tool that games use to not only give you a better ‘sense of place’ but offers you ‘baby interactions’ that prepare you for other parts of the game. I’d love to find some examples of Apps doing this but I honestly can’t find any. If anyone does think of one, please let me know!

APPS
Prompt as little as possible.

GAMES
Prompt all the damn time (they’re just really subtle and sneaky about it).

Lens 5: Hintiness

Hintiness is something completely unique to games. Unlike affordances, which reinforce the current Learning Loop you’re on, the whole point of hintiness is to move you to new loops. Not moving to new loops is a critical problem for games and they have a specific term for it, “Bottom Feeding”.

This screenshot is from World of Warcraft. If you’re a level 3 character and really good at killing spiders that’s great, but when you try to kill something bigger like a crocodile and get killed immediately, most players have a predictable response: go back to killing spiders. This is “bottom feeding”, just taking the easy route and remain at the same level over and never progressing. It’s comfortable, but it’s also boring. It also kills games. Game designers really want you to learn and get better so you stay engaged.

There are all sorts of ways to do this. In Super Mario Odyssey there is a type of navigational hintiness called “Moon Chains”. You’re encouraged to collect these Moon objects in the game but every time you collect one, you can ALWAYS see another one. It hints at where you need to go. 

In Breath of the Wild, the walls are usually smooth stone so when you see one with a broken pattern, it’s a clear sign that something interesting is behind that wall.

But for App UX, hints are exactly the opposite, usually dialog box prompts in your face that stop you dead in your tracks! Games on the other hand, are subtle and usually seen over and over before you finally get it. They are the exact opposite of Microsoft Clippy. That’s part of the discovery and delight. 

The video game approach is just so different from Apps. So many of our users “Bottom Feed” getting stuck in a current simple level of using our apps. If you’ve ever done user testing on your app, you often find that some users will write critical steps down on post-it notes as they keep getting lost. This is a UX tragedy as they don’t understand the Learning Loops well enough so they write down the exact steps every time they need to do something. How can we use hintiness to keep our users from Bottom Feeding?

APPS
Assume users are at a constant skill level.

GAMES
Use hints constantly and patiently to move users to the next level.

EXAMPLE

As there aren’t any good examples in App UX, let me make one up: creating a guide in a drawing app. I’ll use Photoshop, but Sketch or Figma could just as easily use this. There is a “Beginner level” way to create a guide just by using a menu and dialog box:

It’s simple and it works. But most power users know to just drag it out from the bar. How could we use hintiness to help users discover it? One way would be to add a quick, lightweight animation. When the user try the dialog box to add the guide, instead of simply appearing, add an animation ‘sparkle’ in the ruler area and then animate out the guide to the position they asked for. Fast, quick and subtle. The user might not even notice it but after they see it a few times, they might just wonder what was going on and try it themselves. That is what games try to do, get you pulled in and copy what they are showing you.

Lens 6: Pacing

Pacing is critical to the overall story in games but a specific aspect of pacing for games that applies to App UX is “where to start?”. Remember the “Narrative vs Story” section? It was about games creating a tight coupling between experiences in the game to help users create their own narrative:

Games know that a journey of 1000 miles begins with the first step. It’s critical that users know how to get going, get into the flow well before they understand everything. The classic example here would be World of Warcraft, a very complex game that ultimately has dozens of commands for you to learn. But you start with just 1 attack spell. There isn’t much to learn and it starts off pretty easy. As you level up, things are gradually added so you grow into the interface.

What do most Apps do?

They show you EVERYTHING at the very beginning hoping you’ll get it all in this one shot. Of course, we all know what happens: skip, skip, skip….

A colleague of mine at Google, Luke Wroblewski, discussed this at Conversions@Google 2018 where he talked about a finance app that instead of a “help overlay” offered one single clue: how to create an expense item. Of course, the app could do much more but once the user knew how that, they were on their way. 

 Of course, the difficult part is for you to figure out what your “first thing to do” is. This is why games are so interesting, they spend so much time figuring out what we think of as a small, trivial point.

APPS
Tend to offer users a large toolbox and let them figure out how to get started.

GAMES
Have a clear understanding of the journey and say “Start here first”.

Bringing it all home

The six lenses we’ve talked about:

Each provides a window into how games think and design. But it’s not just a simple linear list. The first three lenses are about breaking down your game (or App) into much more concrete chunks:

1. Story vs Narrative (Think in terms of story arcs)
2. Games are fractal (Break up the journey from big to small to tiny)
3. Learning loop (figure out your core mechanic)

And then the following three lenses build things back up:

1. Affordances (Prompt for known loops)
2. Hintiness (Move to new loops)
3. Pacing (Be sure to start here)

It’s really shaped more like this:

These games lenses have been eye opening to me. I feel some of those 80 IQ points Alan Kay promised as I’m thinking about UX design in fundamentally different ways. In this post, I’ve given examples throughout on how this can provide valuable insights in ordinary, every day apps. It’s certainly helped me. Much of my work at Google uses these lenses to break things down fractally, look for loops, and figure out ways to hint at new loops. I’m not building games, but I’m being inspired by them. I hope they inspire you as well.