Data

The use of color to display data is a solved problem, right? Just pick a palette from a drop-down menu (probably either a grayscale ramp or a rainbow), set start and end points, press “apply,” and you’re done. Although we all know it’s not that simple, that’s often how colors are chosen in the real world. As a result, many visualizations fail to represent the underlying data as well as they could.

• one of Research Themes at Ink&Switch
• UI should be just content, everything else directly manipulatable, any buttons and menus take up space that could be there for the actual data

To paraphrase Edward Tufte, too often when we create a data table, we imprison our data behind a wall of grid lines. Instead we can let the data itself form the structure that aids readability by making better use of alignment and whitespace.

In the gif below we start with a table formatted similar to one of Excel's many styling options which, much like the chart styles, do nothing to improve the table. Progressive deletions and some reorganization deliver a clearer and more compelling picture.

As with charts, rather than dressing up our data we should be stripping it down.

I decided to take matters into my own hands and build a cinema selection website I always dreamed of...It’s a website that shows every movie screening in every cinema across the entire Germany. Every screening, every cinema, every movie. All in one long HTML table.

...There’s no logo. No menu. No footer. No pagination. No “See more”. No cookie banners (because no cookies). No ChatGPT/SEO generated bullshit. No ads, of course.

Why? Because people don’t care about that stuff. They care about function. And our UI is a pure function.

• mapping huge spaces onto small screens seems to be one of fundamental problems in "UI"
• somehow free panning in x/y makes me feel like looking through a loupe - I don't get that feeling when scrolling only vertically though
• I'm curious how many and what kinds of projectors would I need to have a wall for doing/thinking

...

• another thing to keep in mind is the "data-ink ratio" idea from Tufte - for power users the best UIs allow for making a lot of information visible at once, with the least amount "UI cruft" possible
• another idea to gain pixel space is to filter out what's irrelevant
• maybe the problem is not so much about the size of the display, but about the DPI? (for reference: reMarkable 2 has 226 DPI, Apple iMac Retina 5k has 218 DPI, so only ~18% of what's possible on paper)

On the advice of McKinsey, Nike’s new CEO John Donahue decided to pivot to a “data-driven” approach, reorganizing the company towards digital direct-to-consumer sales and eliminating the former model centered on distinct categories. The allure is easy to recognize, and it’s the same trap that Boeing and other companies fell into over the preceding years.

Coming up with new ideas is difficult and requires specialist knowledge. Moreover, it requires specialist knowledge to understand what those specialists are doing and therefore manage them.

A useful starting point for designing advanced graphical user interjaces is the Visual lnformation-Seeking Mantra: overview first, zoom and filter, then details on demand. But this is only a starting point in trying to understand the rich and varied set of information visualizations that have
been proposed in recent years. This paper offers a task by data type taxonomy with seven data types (one-, two-, three-dimensional datu, temporal and multi-dimensional data, and tree and network data) and seven tasks (overview, Zoom, filter, details-on-demand, relate, history, and extracts).

Web logs capture traffic metadata, such as the request time and route, how long the server took to respond, the response size, and so on. Analyzing web logs sheds light on both server performance and client behavior. Yet the common practice of summary statistics (e.g., 95th-percentile latency) often hides interesting patterns! This is because performance varies wildly based on the nature of the request, and unusual clients such as bots can easily hide in a sea of “natural” traffic.

What if — instead of summarizing — we plotted every request as a dot with time along x→ and latency (on a log scale) along y↑?

Chernoff faces, invented by applied mathematician, statistician and physicist Herman Chernoff in 1973, display multivariate data in the shape of a human face. The individual parts, such as eyes, ears, mouth and nose represent values of the variables by their shape, size, placement and orientation. The idea behind using faces is that humans easily recognize faces and notice small changes without difficulty. Chernoff faces handle each variable differently. Because the features of the faces vary in perceived importance, the way in which variables are mapped to the features should be carefully chosen (e.g. eye size and eyebrow-slant have been found to carry significant weight).

In the past, GDP and resources use have always been tightly correlated. But this is just drawing a line through some data — it’s not based on any deep theory. And in fact, these correlations can change very quickly. Just as one example, here’s energy use versus GDP since 1949.

If you were sitting in 1970, you could look at this curve and claim, very confidently, that economic growth requires concomitant increases in energy use. And you’d be wrong. Because the trend is your friend til the bend at the end.

Welcome to the Recycle Bin Project.

This is an initiative to keep files in flight.

If you have ever encountered a piece of highly domain-specific business software, you may have noticed that it was largely a graphical front-end to some relational database. You may also, in fact, have avoided using the system at all—studies show that information workers prefer to dump their data into spreadsheets, a general and more familiar tool which, unfortunately, is poorly suited for many standard database tasks. It is time that we stop streamlining the process of creating a new application for every schema, and that we instead develop the visual query languages that will let endusers access the full power of relational database management systems from a simple and unified interface. Once information workers can create, manage, and query real databases with the same ease as they routinely manipulate spreadsheets today, they will never return to their schema-dependent, consultant-made, and oddly-colored Microsoft Access applications.

I opt to proceed by redoing the entire database. It takes a while. Then we rework the Angular app (also very painful) to work with the new (correct) data model. The decision to just do the right thing here instead of limping on forever gave us huge speed benefits later on, and I think it's one of my proudest professional accomplishments even though it sounds simple. It was boring, painful, and no one other than my friend would ever appreciate it. I knew that going in, and I still did it.

Incidentally, I was allowed to do so only because management had been so negligent in allowing things to get to this state that they also didn't notice I was getting them out of this state, and thus couldn't insert their asinine opinions on how good practice is actually too slow. We've got to ship.

I love to tinker with software. It's what got me into vim and Emacs for years. The more I can edit things, install, uninstall, play with config files, the more likely I am to jump in and stay on.

The second thing you need to know is that I'm never happy with the way I work. I’ll take any chance to reduce friction. What are the things that feel like chores, like annoyances, that work against rather than for me? Thus, I’m prone (doomed, even) to every so often question my entire workflow and get excited or distracted by new tools.

And boy, has it been one of those months recently.

Observable Framework is an open-source static site generator for data apps, dashboards, reports, and more. Framework includes a preview server for local development, and a command-line interface for automating builds & deploys.

You write simple Markdown pages — with interactive charts and inputs in reactive JavaScript, and with data snapshots generated by loaders in any programming language (SQL, Python, R, and more) — and Framework compiles it into a static site with instant page loads for a great user experience.

...Framework includes thoughtfully-designed themes, grids, and libraries to help you build displays of data that look great on any device, including Observable Plot, D3, Vega-Lite, Graphviz, Mermaid, Leaflet, KaTeX, and myriad more. (And for working with data, don’t forget about Arquero, DuckDB, and SQLite, too.)

The good old days when America was “great” aren’t the 1950s. They’re whatever decade you were 11, your parents knew the correct answer to any question, and you’d never heard of war crimes tribunals, microplastics or improvised explosive devices. Or when you were 15 and athletes and musicians still played hard and hadn’t sold out.
Not every flavor of nostalgia peaks as sharply as music does. But by distilling them to the most popular age for each question, we can chart a simple life cycle of nostalgia.

We believe this is the most general data structure, able to replace tables, arrays, spreadsheet and relational database, and intrinsically offering built-in visualizations and hands-on controls.
That structure we call zzstructure, or hyperthogonal structure. Like a table it is composed of cells which are connected in rows and at right angles. But there are no overall spatial coordinates.

In a conventional table, the bottom of one cell is connected to the top of another, the left side of once cell is connected to the right side of another.

We generalize this, and say the top of any cell can be connected to the bottom of any cell, and the left edge of any cell can be connected to the right edge of any cell.

...You may think of hyperthogonal structure as:

• sculptures of cells in three dimensions or more
• crossed lists in multiple dimensions
• irregular constructions of cells at right angles and side-by-side
• crystals of lists in corresponding connection

Fits as much data on a screen as possible, while allowing in-depth control of data density to ensure legibility. Pagination is used only when absolutely necessary.

Changing the density-relevant settings Decimals, Group Digits, Min Font Size, Min x-Padding, Min y-Padding does not always lead to a change in UI density, since it is determined by all parameters jointly or, as in the case of Min Font Size, Min x-Padding, Min y-Padding, is already realized since these are minimum values.

A common exercise in the creative computation classes I've taught involves giving students a simple input to see how they can transform it into some visual output. Although I no longer teach, I thought it would be fun to challenge myself with a similar prompt. My input is the current time, and my output needs to incorporate data.

This problem of long-term digital storage seemed a crucial hurdle for any civilization trying to act generationally. How could a society think in terms of centuries unless there was a reliable way to transmit and store its knowledge over centuries? This puzzle was the focus of a conference hosted by Long Now in 01998, dedicated to technical solutions for Managing Digital Continuity. At this meeting Brewster Kahle of the Internet Archive suggested a new technology developed by Los Alamos labs, and commercialized by Norsam Technologies, as a solution for long term digital storage. Norsam promised to micro-etch 350,000 pages of information onto a 3-inch nickel disk with an estimated lifespan of 2,000 -10,000 years.

• the right interface can have an incredible power

  • making something that was previously extremely hard, available to everyone (switching from Roman to Arabic numerals as an interface to mathematics)
  • making invention possible (Feynman Diagrams)
  • making Solving Things Visually possible (Cartesian Coordinates)
    • Data Visualization can be thought of as an alternative, sometimes more powerful, interface to data
  • working with the amounts of data for which we don't have enough Pixel Space (Collection-Browsing Interfaces)

• new powerful notations "stick around" and infect human thinking for generations

When I polled my community about their attitudes toward busywork — a ruse to figure out what some of my nearest and dearests actually do for work — most at least saw value, if not joy, in occasional busywork. A web designer told me busywork serves as “productive procrastination” when she’s avoiding more complex tasks. A woman in sales and marketing said she values the solitude of rote tasks, and retreats into spreadsheets “when everybody’s annoying and I’m peopled out and my bullshit meter is filled.” A senior research program manager at a nonprofit explained that she values how data cleaning — combing through a dataset for errors, duplicates, and other issues — creates an intimacy with the information she’s processing. Cleaning data manually makes the phenomena she studies less abstract: “It connects you to a different way of working or being, or creates opportunities to see things in a different way.”

File over app is a philosophy: if you want to create digital artifacts that last, they must be files you can control, in formats that are easy to retrieve and read. Use tools that give you this freedom.

File over app is an appeal to tool makers: accept that all software is ephemeral, and give people ownership over their data.

Back in 2019, I started collecting all kinds of metrics about my life. Every single day for the last 3 years I tracked over 100 different data types - ranging from fitness & nutrition to social life, computer usage and weather.

Naturally after I started collecting this data, I wanted to visualize what I was learning, so I created this page

It's really common to see claims that some meme is backed by “studies” or “science”. But when I look at the actual studies, it usually turns out that the data are opposed to the claim. Here are the last few instances of this that I've run across.

Miners seek valuable nuggets of ore buried in the earth, but have no control over what is out there or how hard it is to extract the nuggets from their surroundings. ... Similarly, data miners seek to uncover valuable nuggets of information buried within massive amounts of data.

Farmers cultivate the land to maximize their yield. They manipulate the environment to their advantage using irrigation, pest control, crop rotation, fertilizer, and more. Small-scale designed experiments let them determine whether these treatments are effective. Similarly, data farmers manipulate simulation models to their advantage, using large-scale designed experimentation to grow data from their models in a manner that easily lets them extract useful information.

Virtually every user in the study reported that an advantage of spreadsheets is the ability to view large quantities of data on the screen (Nardi and Miller, 1990). Users had a strong preference for being able to view as much data as possible without scrolling.

Humans love symmetry. We find symmetry to be very attractive. Our brains may even be hard-wired through evolution to process symmetrical data more efficiently. So, it's no surprise that, as designers, we try to build symmetry into our product interfaces and layouts. It makes them feel very pleasant to look at.

Unfortunately, data is not symmetrical…Once you release a product into "the real world", and users start to enter "real world data" into it, you immediately see that asymmetrical data, shoe-horned into a symmetrical design, can start to look terrible.

To fix this, we need to lean into an asymmetric reality. We need to embrace the fact that data is asymmetric and we need to design user interfaces that can expand and contract to work with the asymmetry, not against it. To borrow from Bruce Lee, we need to build user interfaces that act more like water:

“You must be shapeless, formless, like water. When you pour water in a cup, it becomes the cup. When you pour water in a bottle, it becomes the bottle. When you pour water in a teapot, it becomes the teapot. Water can drip and it can crash. Become like water my friend.” — Bruce Lee

[We observed that] most people just used Excel to make lists. Suddenly we understood why Lotus Improv, which was this fancy futuristic spreadsheet that was going to make Excel obsolete, had failed completely: because it was great at calculations, but terrible at creating tables, and everyone was using Excel for tables, not calculations.

Bing! A light went off in my head.

The great horizontal killer applications are actually just fancy data structures.

Spreadsheets are not just tools for doing “what-if” analysis. They provide a specific data structure: a table. Most Excel users never enter a formula. They use Excel when they need a table. The gridlines are the most important feature of Excel, not recalc.

Word processors are not just tools for writing books, reports, and letters. They provide a specific data structure: lines of text which automatically wrap and split into pages.

PowerPoint is not just a tool for making boring meetings. It provides a specific data structure: an array of full-screen images.

Many AI products today are focused on indexing as much as possible.

...But more information isn’t always better. The limits of the ‘index everything approach’.

Index size is a trade-off against retrieval quality. A larger index can capture more information, but it also increases the risk of false positives in retrieval.

If someone gives you a CSV file with 100,000 rows in it, what tools do you use to start exploring and understanding that data?

4.2 gigabytes.

That’s the size of the model that has made this recent explosion possible. 4.2 gigabytes of floating points that somehow encode so much of what we know.

...There is already much talk about practical uses. Malicious uses. Downplaying. Up playing. Biases. Monetization. Democratization - which is really just monetization with a more marketable name.

I’m not trying to get into any of that here. I’m just thinking about those 4.2 gigabytes. How small it seems, in today’s terms. Such a little bundle that holds so much.

An API should be easy to learn and write to, and hard to misuse. Your API will also need to evolve, and a good design takes this into account.

One common element of the larger places where I've worked is that they tend to have a directory service of some sort that keeps track of who's an employee and who isn't. You can learn some interesting things by periodically dumping that list and then running comparisons against the previous dump.

...Incidentally, if someone gets mad about you running this sort of thing, you probably don't want to work there anyway. On the other hand, if you're able to build such tools without IT or similar getting "threatened" by it, then you might be somewhere that actually enjoys creating interesting and useful stuff. Treasure such places. They don't tend to last.

Without a person at (or near) the helm who thoroughly understands the principles and elements of Design, a company eventually runs out of reasons for design decisions. With every new design decision, critics cry foul. Without conviction, doubt creeps in. Instincts fail. “Is this the right move?” When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions.

Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle.

A website that provides a comprehensive list of items that can be easily copied and pasted. From text to images to HTML codes and much more, the website offers a wide variety of items that can be used for almost any purpose.

SQLime is an online SQLite playground for debugging and sharing SQL snippets.