Interaction

From transactions to meetings, every event happens at a point or duration in time. Successful investigations need to understand how those events unfold, and how they’re linked.

KronoGraph is the first toolkit for scalable timeline visualizations that reveal patterns in time data.

• one of Research Themes at Ink&Switch
• UI should be just content, everything else directly manipulatable, any buttons and menus take up space that could be there for the actual data

My problem [with Microsoft's Productivity Future Vision], from an interaction perspective, is not visionary. It's a timid increment from the status quo, and the status quo, from an interaction perspective, is terrible.

This matters, because visions matter. Visions give people a direction and inspire people to act, and a group of inspired people is the most powerful force in the world. If you're a young person setting off to realize a vision, or an old person setting off to fund one, I really want it to be something worthwhile. Something that genuinely improves how we interact.

We don't necessarily need to constantly interact with people “around” us on the web. The sensation of being in the quiet companionship of someone else, like reading next to them in a cafe, is what we're missing. The sense of ambiently sharing space – of being co-present – while engaged in other activities is a staple of shared public spaces that we're still figuring out how to design in the digital realm.

The most powerful way to gain insight into a system is by moving between levels of abstraction. Many designers do this instinctively. But it's easy to get stuck on the ground, experiencing concrete systems with no higher-level view. It's also easy to get stuck in the clouds, working entirely with abstract equations or aggregate statistics.

This interactive essay presents the ladder of abstraction, a technique for thinking explicitly about these levels, so a designer can move among them consciously and confidently.

Android and iOS share a common problem: they copied desktop text editing conventions, but without a menu bar or mouse. This forced them to overload the tap gesture with a wide range of actions: placing the cursor, moving it, selecting text, and invoking a pop-up menu. This results in an overly complicated and ambiguous mess-o-taps, leading to a variety of user errors.

It’s less of a problem if you only do short bursts of text in social media or messaging apps. But doing anything more complicated like an email gets tedious. However, in my user study on text editing, I was surprised to find that everyone had significant problems and rather severe workarounds for editing text.

My design goal was simple: A person can walk up, play with a MIDI controller, see a resulting image on a screen, and then send the image to a pen plotter. The plotter draws their unique drawing, and they can take it home and frame it.

...Typically, Generative Art is fully autonomous, and a pseudo-random number generator is called to create a randomized set of parameters. For AAS, I intended to pluck out the randomness and insert user controls.

I evaluated a few options for input devices. I landed on the Intech Grid series MIDI controllers for their simplicity and modularity. Meant to be snapped together, these controllers offered the correct number of inputs to encourage playful exploration without being overwhelming.

People are alive -- they behave and respond. Creations within the computer can also live, behave, and respond... if they are allowed to. The message of this talk is that computer-based art tools should embrace both forms of life -- artists behaving through real-time performance, and art behaving through real-time simulation. Everything we draw should be alive by default.

Part 1 talks about the potential of the computer as a new visual art medium. I show a collaboration between art and artist, with the art behaving through simulation, the artist behaving through performance, and the two of them working together, responding to each other.

Part 2 demonstrates a tool for "programming" how art should behave and respond. The artist directly manipulates art objects on the canvas, the way that visual art has always been created since the time of cave paintings. The tool is based around direct, geometric construction rather than indirect, algebraic "code".

Unfolding functionality
MouseOver
MouseDragging instead of Value-Slider
Memory Dots / Five Favorites
Transparency & Shadows
Full Screen Mode, Rooms Metaphor
Workspace – Desktop
MetaWindow

Unit is a general-purpose visual programming system built for the future of interactivity.

All-in-one / Tactile / Sandboxed / 0 Setup / Auditable / Reprogrammable / Structured Editing / Meta-Programming / Private / Full Stack / Code + Layout / Open Source / Components / Reactive / Multimodal / Shareable / General / No Context Switch / Typed / Ergonomic / Handcrafted / Live / Customizable / Explorable / Visual / Dataflow / Expressive / Multiplayer / Physical / Mobile

It’s easy to move the cursor from Amazon’s main dropdown to its submenus. You won’t run into the bootstrap bug. They get away with this by detecting the direction of the cursor’s path.

At every position of the cursor you can picture a triangle between the current mouse position and the upper and lower right corners of the dropdown menu. If the next mouse position is within that triangle, the user is probably moving their cursor into the currently displayed submenu. Amazon uses this for a nice effect. As long as the cursor stays within that blue triangle the current submenu will stay open. It doesn’t matter if the cursor hovers over “Appstore for Android” momentarily – the user is probably heading toward “Learn more about Cloud Drive.”

And if the cursor goes outside of the blue triangle, they instantly switch the submenu, giving it a really responsive feel.

Situated software isn't a technological strategy so much as an attitude about closeness of fit between software and its group of users, and a refusal to embrace scale, generality or completeness as unqualified virtues. Situated software…is personal from its inception. Teachers on the Run worked this way. Everyone knew that Paul and Keren built it. You could only rate Clay and Marianne and Tom and the other ITP professors. You didn't even know it even existed unless you were on the ITP mailing list. The application's lack of generality or completeness, in other words, communicated something – "We built this for you" – that the impersonal facade of RateMyProfessors.com doesn't have and can't fake.

Both patterns are easy to use and work on mobile.

But there’s an even more important lesson here.

Because Victor was thoughtful, no doubt. He looked at the slider, spotted an issue and came up with better UX.

But improving a bad pattern doesn’t make it good. You’re usually better off finding another pattern altogether.

Let me argue that the actual dawn of user interface design first happened when computer designers finally noticed, not just that end users had functioning minds, but that a better understanding of how those minds worked could completely shift the paradigm of interaction.

This is called undesirable correlation. It looks like left panel (Detect language) asks you to drag and drop and right one (English) to choose a file.

In reality, both actions apply to the left one! Or to the entire UI, if you will.

Anyway, try to avoid.

To design a system means to orchestrate the interplay of its elements.

Such a system is considered “interactive” if it is open, which means that there are ways to engage with the processes that are happening inside of it. There is of course a range of interactivities which spans from very basic reactive behaviour to highly complex conversational interactions.

You’ve all seen them, tiny switches that let you toggle a setting. And maybe, just like me, you sometimes pause, thinking “…Is it on or off?” That’s because toggles suck!

...Just use a checkbox or radio group! It’s quick to implement, accessible from the start, and doesn’t make the user think.

It’s a game we’d been dreaming of for years: a giant rube goldberg machine builder in the style of the classic Incredible Machine games, made of a patchwork of machines created by individual xkcd readers.

...Sometimes an idea feels like it emerges fully-formed, but when you start talking about it, you realize there’s still a dizzying array of decisions to make. Thus ensued 5 days of brainstorming to discover each of us had slightly different core beliefs about what this comic should be:

• Where do the balls come from?
• Does everyone see the same machine? What is its purpose?
• How can players interact with it?
• And most importantly… why do they?

One of the great failures of modern computing is how it has largely ignored the presence of the human body beyond the slightest acknowledgement that humans have a pair of eyeballs and a few fingertips.

...Consider, for instance, our hands. Human hands are capable of so much more than typing and swiping. Human finger sensitivity is astonishingly precise, capable of detecting wrinkles on a smooth surface with a wavelengths as small as 10nm, features so small they are invisible to the eye. Hands can both manipulate and communicate, have effect on the world and perceive its most minuscule features.

Enthusiasm around large language models (LLMs) and text-to-image systems, however, has made it easy to forget about the impressive facility of our hands and the astonishing capabilities of our vision systems, leading instead to proclamations about the pending death of the graphical user interface as an input system. Instead of veering towards a future where computers make greater use of our bodies’ expressiveness, LLMs compel us to diverge further from such a course. Why manipulate anything on a computer with gesture or point and click when you can type or speak?

WELCOME
TO
THE
WORLD
WIDE
WEB,
AN
INTERACTIVE
MEDIUM
IN
WHICH
SCREEN
RESOLUTION
STATISTICS
ARE
TRIVIAL,
BROWSER
VIEWPORTS
ARE
VARIABLE,
AND
SCROLLING
BEHAVIOUR
IS
A
STANDARD.

The idea is we want navigation links at some default level of color contrast. On hover, two different actions are triggered which results in the hovered link gaining more color contrast while the rest of the links lose additional contrast. It all adds up to something neat where we’re both emphasizing the hovered link and de-emphasizing the others in a single interaction.

Sometimes, when you don’t work on a feature idea right away, it loses steam and fades away. Yet, the idea of multi-edit remained surprisingly popular at Figma.

...Multi-edit is the feature which has had the longest gap between initial idea and launch. Over those years, we kept picking it up and putting it down, and we kept polishing and iterating on it. It feels like a rock that we have been tumbling in a rock tumbler until it is shiny and smooth.

One of the odd things about this process is that as we refined multi-edit, it almost started fading away. We’ve been living with multi-edit internally for months now, and we’ve kind of stopped noticing it—until we switch back to the production environment, and we realize how inefficient things were before!

In the proposed interface, with a simple hover action I receive essential pieces of information. I am able to understand my thought process and points of interest. It is also easier to actually remember the page thanks to particular color schemes and meta sections.

Our physical bodies play a central role in shaping human experience in the world, understanding of the world, and interactions in the world. This paper draws on theories of embodiment—from psychology, sociology, and philosophy — synthesizing five themes we believe are particularly salient for interaction design: thinking through doing, performance, visibility, risk, and thick practice.

It’s been 19 years since Pixar released Monsters, Inc. with all that CGI hair. Where are my hairy icons? Ones that get all long and knotted as the notifications number goes up.

Why can’t I feel my phone? I found that paper from 2010 (when I was complaining about keyboards) about using precision electrostatics to make artificial textures on touchscreens.

I should be able to run my thumb over my phone while it’s in my pocket and feel bumps for apps that want my attention. Touching an active element should feel rough. A scrollbar should *slip. Imagine the accessibility gains. But honestly I don’t even care if it’s useful: 1.5 billion smartphone screens are manufactured every year. For that number, I expect bells. I expect whistles.

That means that every visual choice you make — layout, colors, typography, images — must be focused on getting a distracted, rushed, and emotionally-driven brain through their initial sub-conscious vetting and into the kind of systematic processing — detailed and focused review — you assume they’ll do initially.

In 2014, I wrote about my belief that design and engineering are best when tightly woven together. That’s truer now than ever.

If I’m feeling confident, I’ll jump right into my text editor…From here, more functionality is added and the code is tweaked until the feature looks and feels right to me. Whether it’s something simple like this, or prototyping a new interaction like multi-connect, there’s no substitute for designing with real code.

In rare cases when I have ideas or plans that I’m less confident about, it’s time to break out the paper, pens, and markers,

Because the Kinopio interface elements and aesthetic are full-grown, I almost never use traditional design software anymore.

This first-generation Vision Pro hardware is severely restricted by the current limits of technology. Apple has pushed those limits in numerous ways, but the limits are glaring. This Vision Pro is a stake in the ground, defining the new state of the art in immersive headset technology. But that stake in the ground will recede in the rear view mirror as the years march on. Just like the Mac’s 9-inch monochrome 512⁠ ⁠×⁠ ⁠342 pixel display. Just like the iPhone’s EDGE cellular modem.

But the conceptual design of VisionOS lays the foundation for an entirely new direction of interaction design. Just like how the basic concepts of the original Mac interface were exactly right, and remain true to this day. Just like how the original iPhone defined the way every phone in the world now works.

We can imagine, for example, mechanisms by which a user debugs a particular formula by bringing up a view of the spreadsheet in which irrelevant rows and columns are filtered out, and related cells are highlighted.

Speedometer is a browser benchmark that measures the responsiveness of Web applications. It uses demo web applications to simulate user actions such as adding to-do items.

An ex-Apple interface designer’s 40-year plan to redesign not just the way we use computers, but the way we think with them.

Reminds me of a little feature I like in Notion where if you type dash-arrow (like ->) it turns into → — but intelligently — like it doesn’t do that with inline code or a code block.

When I first experienced this demo back in June at WWDC, reps from Apple said — half boasting, half warning (both halves well-warranted) — that when the butterfly lands on your finger, some people actually feel it. They feel a physical tickle on their finger...When an illusion is realistic enough, your brain accepts it as real, even though it’s not — and even though you know it’s not.

...The experience of manipulating the user interface in VisionOS conveys something like this too...One comparison would be to a maestro conducting an orchestra. But that doesn’t convey the sense of precision in VisionOS — the sense of fine control through indirect manipulation. What it feels like is using the Force.

Design work will look very different in the future. It’s inevitable, given the pace of technological change. How can you prepare for the future, whatever it holds? The answer is that you need to think differently about your work.

• Embrace Change
• Ground Work in the Timeless
• Adopt Models as the Object of Design

Downpour is the best way to make games on your phone. Collage together photos, drawings and text, and then connect them into an interactive story. It's genuinely quick and easy to use — you can make a game before your tea has gone cold.

Key Event Viewer: View KeyboardEvents on a <input type="text"> element.

Mouse Event Viewer: View MouseEvents on a set of overlapping <div>s.

Focus Event Viewer: View FocusEvents between two <input> fields enclosed in a <div>.

.button,
::file-selector-button {
  inline-size: fit-content;
  touch-action: manipulation;
  user-select: none;
}
* {
  &:focus-visible {
    outline: 2px solid magenta;
    outline-offset: 2px;
  }
}

Users really care about speed in interaction design...A snappy user experience beats a glamorous one, for the simple reason that people engage more with a site when they can move freely and focus on the content instead of on their endless wait.

• 0.1 seconds gives the feeling of instantaneous response. This level of responsiveness is essential to support the feeling of direct manipulation.
• 1 second keeps the user's flow of thought seamless.
• 10 seconds keeps the user's attention. A 10-second delay will often make users leave a site immediately.