Research

Annotation tools feel great to use, but they don’t support the follow-up thinking you need to do. Text editors give you a flexible canvas for making sense of snippets, but their design is often cumbersome and disorienting when used in this way. If you could move fluidly between these tools, you could use each where it excels and, perhaps, get the best of both worlds. In this paper, we present Latticework, a system which unifies annotation with freeform text editing, in the context of personal knowledge management tools.

[In Latticework], you forage through messy source documents, accumulating key snippets into a working document for sensemaking (i.e. rearranging and elaborating that material for insight). But, as prior work has described, this process isn’t linear. It’s often convenient to do a bit of preliminary sensemaking in the midst of foraging; conversely, observations you uncover during sensemaking will often lead to another round of foraging, and so on, in a loop.

Latticework’s main goal, then, is to enable fluid movement between these foraging and sensemaking stances. By extension, that means fluid movement between acting on source documents (which emphasize foraging) and on your working document (which emphasizes sensemaking). Ideally, you should be able to shift your focus as it makes sense in the moment, and the work you do in each place should remain visible in the other.

Cartographist is an experimental web browser optimized for rabbit-holing.

• Instead of opening new windows (with cmd-click), Cartographist spawns horizontally scrollable panes.
• Instead of forcing you to find things in a linear history, Cartographist shows a tree-structured outline of your browsing.
• Instead of always starting fresh, Cartographist can save, and load "trails" — the exact state of the session you've left — supporting researching topics over long periods of time.

The existing personal knowledge management tools are insufficient to help us process information, especially during web-based research.

The tree structure and unique file path inherited from analog metaphors encourage collection, not connection, of ideas.

They make it harder to reuse and cross-reference ideas, and easier to hoard information.

...Can building associations be more intuitive?
...Is playfulness possible?
...Can web discovery be an organic extension of the existing knowledge base?

Semilattice is a collection of system and interaction concepts for personal knowledge management tools.

As we’ve been researching what design teams need to do to create great user experiences, we’ve stumbled across an interesting finding. It’s the closest thing we’ve found to a silver bullet when it comes to reliably improving the designs teams produce.

The solution? Exposure hours. The number of hours each team member is exposed directly to real users interacting with the team’s designs or the team’s competitor’s designs. There is a direct correlation between this exposure and the improvements we see in the designs that team produces.

Imagine a circle that contains all of human knowledge.
By the time you finish elementary school, you know a little.
By the time you finish high school, you know a bit more.
With a bachelor's degree, you gain a specialty.
A master's degree deepens that specialty:
Reading research papers takes you to the edge of human knowledge.
Once you're at the boundary, you focus.
You push at the boundary for a few years.
Until one day, the boundary gives way.
And, that dent you've made is called a Ph.D..
Of course, the world looks different to you now.
So, don't forget the bigger picture.
Keep pushing.

If large groups are not ideal, what is the perfect group size? A fascinating 2019 study explored this question by looking at citation networks. They found that papers with more authors tend to receive more citations — large teams are good at developing a field. However, they found that the smallest teams — between one and three authors — were significantly more likely to publish disruptive results that could change the course of a field. So, in terms of sheer creativity, smaller groups seem to have an advantage.

With three or more people, group think and social dynamics kick in; there is an audience to impress. Thus, the ideal group may actually be of a minimal size: two. When working with just one other person, one must remain fully focused as the pair iteratively move the discussion forward. Two people who support each other's thinking can travel far in their thinking without getting distracted. With just one other person, it is also easier to be at ease and to enjoy the experience — to get into a state of 'flow'.

On the advice of McKinsey, Nike’s new CEO John Donahue decided to pivot to a “data-driven” approach, reorganizing the company towards digital direct-to-consumer sales and eliminating the former model centered on distinct categories. The allure is easy to recognize, and it’s the same trap that Boeing and other companies fell into over the preceding years.

Coming up with new ideas is difficult and requires specialist knowledge. Moreover, it requires specialist knowledge to understand what those specialists are doing and therefore manage them.

Conventional tech industry product practice will not produce deep enough subject matter insights to create transformative tools for thought.

...The aspiration is for any team serious about making transformative tools for thought. It’s to create a culture that combines the best parts of modern product practice with the best parts of the (very different) modern research culture. You need the insight-through-making loop to operate, whereby deep, original insights about the subject feed back to change and improve the system, and changes to the system result in deep, original insights about the subject.

Instead of spending time planning and concocting roadmaps, replace that activity by talking to current or potential customers on how their lives can be improved, and letting that determine your next feature. Injecting the actual customer who uses the software into the development process is key to creating value. The more proxies you have between the developer and the customer, the less the product will meet customer needs.

There is no point to having a large backlog because the bigger the backlog, the higher the unvalidated assumptions, and the lower the chance that it creates any customer value. I have made too many mistakes assuming that something is valuable, when nobody cares about it. A large backlog should be looked at with an extremely high degree of skepticism, as the size of your backlog is inversely proportional to how often you talk to customers.

Planning time is best used focusing on how to build the feature, not what to build. The what should come directly from the customer, with the software development process needing a direct physical line to customer - no proxies.

There are three types of work that UX Researchers need to do:

• Macro-research is strategic in nature, business-first, and future-thinking. It provides concrete frameworks that guide macro business decisions.
• Middle-range research is focused on user understanding and product development.
• Micro-research is closer to technical usability, eye tracking, and detailed interaction development.

The biggest reason UX Research is facing this reckoning is that we do way, way too much middle-range research.

Middle-range research is a deadly combination of interesting to researchers and marginally useful for actual product and design work. It’s disproportionately responsible for the worst things people say and think about UXR. Doing so much of it just doesn’t deliver enough business value.

The feeling of fortuitous gratitude at coming across unexpected information is something most of us who’ve done any research, have experienced — that kismet of finding the perfect book, one spine away from the one that was sought. In the field of art and image research, this sparking of transmission, of sequence and connection, happens on a subconscious level.

…Why is the vernacular image still being dismissed as ephemera? Why is its study not being prioritized? All languages are alive, but visual language is galactic. Keywords are not eyeballs, and creating rutted pathways to follow is the antithesis of study. A century of visual language, knowledge, and connectivity is marching toward a narrow, parsimonious basement of nomenclature. The NYPL takes a step backward if it models its shelves and research on a search engine. Spontaneity is learning. Browsing is research.

Rewind is a personalized AI powered by everything you’ve seen, said, or heard.

Stories like mine are why we need to fund “basic” or “curiosity-based” research. Governments love to pretend that we can do science more efficiently by only funding work that addresses a well-defined economic or societal need – but expecting such a strategy to work is monumentally foolish. Everyone knows a story like mine. It might be a famous one, like the accidental invention of Post-It notes; or it might be more obscure, like the accidental discovery of phosphorus by a gold-seeking alchemist. And many such stories haven’t finished unfolding yet: it may take years for the accidental importance of one’s work to become clear. Science has advanced this way for hundreds of years, with smart people following their curiosity around the twists and turns that nature throws at them.

Actually, in a larger sense accidental discoveries aren’t accidental at all. They may be individually unpredictable, but in the aggregate, they’re the fruits of a “let’s unleash curiosity” strategy. We should tell our accidental stories often, because society needs to know that there’s no better way to move science forward than to unleash the smart and curious, and wait for “accidents” to happen.

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.

Given how quickly concept maps help teams get aligned on their understanding, I thought it might be useful to develop a tool to expedite the creation of concept maps. I don’t mean a tool to help you draw a map. After all, there are great diagramming tools in the market. I mean a tool that will draw concept maps for you.

Synthesis is one thing LLMs do well. Ask ChatGPT to summarize an article, and you’ll get a useful couple of paragraphs that get to the gist of whatever you’re wanting to understand. But the output is still text. What if it could do the same but render a diagram instead?

This talk centered on Hamming's observations and research on the question "Why do so few scientists make significant contributions and so many are forgotten in the long run?"

As Nosrat provides a simple list of essential ingredients for any great meal, can we describe a simple list of essential components for digital products?

Here are four elements that I believe are the foundation of great digital products: Research, Empathy, Simplicity and Speed.

There is a hidden cost to having a hypothesis. It arises from the relationship between night science and day science, the two very distinct modes of activity in which scientific ideas are generated and tested, respectively [1, 2]. With a hypothesis in hand, the impressive strengths of day science are unleashed, guiding us in designing tests, estimating parameters, and throwing out the hypothesis if it fails the tests. But when we analyze the results of an experiment, our mental focus on a specific hypothesis can prevent us from exploring other aspects of the data, effectively blinding us to new ideas.

Scientific papers straddle two worlds. They’re thoughtfully crafted prose documents, but they’re also computational documents containing data analyses and visualizations. Today, the prose and computational parts of a paper often live in different environments and tools, which causes friction for teams of scientists.

...We wondered: what if one collaboration environment could host both the text of the paper and the data visualization code, making it seamless to edit them together?

The demo shows a web-based collaboration environment with provenance — information about how computed artifacts were generated from source material. By keeping track of provenance, we know when an output file needs to be rebuilt, and we know how to do it. We can also use provenance to create a map of the project.

Thinking time — the time needed to concentrate without interruptions has always been central to scholarly work. It is essential to designing experiments, compiling data, assessing results, reviewing literature and, of course, writing. Yet, thinking time is often undervalued; it is rarely, if ever, quantified in employment practices.
Newport's thesis raises a much more fundamental question: what is the impact of lost concentration time on science — not just on the structure and process of science, but also on the content and quality of research?

...Felicity Mellor, a science-communication researcher at Imperial College London, is sceptical about giving managers a role in thinking time. In many cases, researchers are already feeling the weight of their institution's monitoring and evaluation systems. Mellor argues that including yet another box in an evaluation form might not go down well. She also thinks that institutions will not accept this. "Can you imagine the response if a scientist filled out a time sheet where it says 'eight hours spent thinking'?" Ultimately, she says, creating a more supportive research culture needs a much more fundamental change. That suggests an even more radical rethink of the current funding model for academic research.

...[Cal] Newport's thesis [in Slow Productivity] raises a much more fundamental question: what is the impact of lost concentration time on science — not just on the structure and process of science, but also on the content and quality of research?

Million Short makes it easy to discover sites that just don't make it to the top of the search engine results for whatever reason – whether it be poor SEO, new site, small marketing budget, or competitive keywords. The Million Short technology gives users access to the wealth of untapped information on the web.

Being an end-user of someone else’s field notes certainly gives you insight into the benefits of good note-taking skills. Our experiences as end-users and creators of archival field notes lead us to a few specific recommendations:

(1) Don’t get bogged down in the details of format or style.

Rules are counterproductive if they prevent a researcher from taking field notes in the first place.

You will get more return by focusing on your content than by refining your formatting.

(2) Compose your notes as if you were writing a letter to someone a century in the future.

Writing for an external audience requires you to be more explicit in your descriptions and to take less knowledge for granted. Avoid the use of abbreviations, symbols, and other shortcuts that only you will understand.

Ask yourself: How would you describe this to someone over the phone?

(3) It is better to spend five minutes writing the important details than twenty minutes writing the trivial ones.

As a banana who lives among humans, I am naturally interested in humans, and in the social sciences they use to study themselves. This essay is my current response to the Thiel question: “What important truth do very few people agree with you on?” And my answer is that surveys are bullshit.

...The ordinary survey, in its ordinary conditions, is of no evidentiary value for any important claim. Just because there exist rare conditions where survey responses tightly map to some condition measurable in other ways does not mean that the vast majority of surveys have any value.

It's really common to see claims that some meme is backed by “studies” or “science”. But when I look at the actual studies, it usually turns out that the data are opposed to the claim. Here are the last few instances of this that I've run across.

"How often do you..." vs "How often would you..." vs. "Last week did you?". The responses to those could be wildly different. Did you check your words as closely as you checked your results?

If you slide back a few steps, you'll see that it's all a judgement call. How you asked, what you asked, when you asked, which words you paired together, how you started the question... All these things matter. In fact, they are the matter that form the answer.

And you know what? There's no right way. Which is the exactly the point I'm trying to make.

Your hard data comes from subjectivity. A sense of solidity built from mush. False confidence at the finish line, from a shapeshifting starting line.

Metrics come up when we’re talking about A/B testing, growth design, and all of the practices that help designers get their seat at the table (to use the well-worn cliché). But while metrics are very useful for measuring design’s benefit to the business, they’re not really cut out for measuring user experience.

Elicit uses machine learning to help you with your research: find papers, extract key claims, summarize, brainstorm ideas, and more.

Computers can aid humans in our most noble endeavors: art, science, thinking, self-improvement. But today’s dominant computing platforms increasingly work against the needs of creative professionals. Ink & Switch is an independent research lab working on this problem.

In a world with over 5 million podcasts, Dwarkesh Patel stands out as an unexpected trailblazer...of all the noteworthy parts of Patel’s journey to acclaim, one thing stands out among the rest: just how deeply Patel will go on any given topic.

...This level of deep research is, for him, a way of respecting the opportunities he has in being able to speak to highly-regarded figures with influence, despite starting from a place of little influence himself.

Writing does not come particularly naturally to me, and my process reflects that.

There are, broadly, two types of posts I write (with significant overlap). The first are explanation-driven posts, where I try to explain some given topic. More often than not, the impetus is to explain something to myself: I feel like I don’t understand a topic, and use writing as a way to work through an understanding of it.

...The second type of post I write is exploratory, rather than explanatory. I find some dataset or collection of information that I think will be interesting to explore, and I write up what I find while exploring it.

...These categories will often overlap: I’ll be interested in understanding something, and do so by exploring some dataset (or multiple datasets).

It would be great if the U.S. had a giant company that could do for energy tech what Bell did for electronics and Google did for AI. But how could such a company be willed into existence? Perhaps the U.S. could create a national private electrical company.

...A key feature of Energy Bell would be price controls on the electricity the company provided.

...A second requirement for Energy Bell would be that it would have to create its own equivalent of Bell Labs, and spend a certain amount of money on research.

I am inclined to believe that the physical aspect of reading a printed book is a significant benefit in one’s retention of its content, but I’m not aware of any proof of this: research is slight and rather ambiguous, suggestive rather than probative. I also believe that writing something out by hand forces you to work at a speed similar to that of careful thought and that this improves your prose — but can anyone prove that? What I do believe/know is that we have not been reading or writing for long enough for any kind of evolutionary changes to have taken place in our brains to make one format obviously better than any other format.

Imagine the situation. You’re at home, eating dinner. You get a call from someone who says “Hello, this is Public Policy Polling. Would you mind answering some questions for us?” You say “Sure”. An extremely dignified sounding voice says – and this is the exact wording of the question – “Do you believe that shape-shifting reptilian people control our world by taking on human form and gaining political power to manipulate our society, or not?” Then it urges you to press 1 if yes, press 2 if no, press 3 if not sure.

So first we get the people who think “Wait, was 1 the one for if I did believe in lizardmen, or if I didn’t? I’ll just press 1 and move on to the next question.”

Then we get the people who are like “I never heard it before, but if this nice pollster thinks it’s true, I might as well go along with them.”

Then we get the people who are all “F#&k you, polling company, I don’t want people calling me when I’m at dinner. You screw with me, I tell you what I’m going to do. I’m going to tell you I believe lizard people are running the planet.”

And then we get the people who put “Martian” as their nationality in psychology experiments. Because some men just want to watch the world burn.

Do these three groups total 4% of the US population? Seems plausible.

That Caro’s work is still done on paper, with no digital backup to speak of, marks him as one of the last of his kind. (He had never seen a Google doc until I offered to show him one. He was mildly startled to discover that, in a shared document, the person on the other end can be seen typing in real time: “That’s amazing. What’s it called? A doc?”) The Society has his old Smith-Corona Electra 210 on display, but he’s hung on to a bunch of duplicate models and a large quantity of black cotton typewriter ribbons so he can continue to work the way he always has. He handwrites first, then types it up, triple-spacing in the old newspaper fashion, then pencil-edits and retypes, pencil-edits and retypes.

Shock absorption in woodpeckers is a good example of how hypotheses can spread to become common beliefs even with no scientific evidence supporting them.

The hardest thing about customer interviews is knowing where to dig. An effective interview is more like a friendly interrogation. We don’t want to learn what customers think about the product, or what they like or dislike — we want to know what happened and how they chose... To get those answers we can’t just ask surface questions, we have to keep digging back behind the answers to find out what really happened.

"The Mom Test" book explains the solution - it’s about framing your questions in such a way that you get truthful, unbiased feedback, even from those who are inherently supportive, like your mother.

...The key is to ask questions that dig deeper, inquiring about actual behavior, like whether someone has ever actively sought a solution like yours.

Enough feedback comes quick.

Invite 100, get 10, 5 probably tell you all you need to know. Or at least 80% of it, which is all you need to know.

Another 10, 20, 50+ will end up telling you mostly the same as those core five.

A/B testing is an effective approach to use science to design and deliver deeply-frustrating user experiences.

A/B testing without upfront research is just random monkeys testing random designs to see which of those designs do “best” against random criteria.

If drug testing was actually implemented like most A/B tests, you’d give 2 drugs to 2 groups of people and pick the “winner” by whichever group had fewer deaths.

Voyant Tools is a web-based reading and analysis environment for digital texts.

Ethnographic studies are distinct from ethological research in other species because we can speak with our subjects and ask them questions. This has tremendous value, but much of what humans do is not spoken, and we also observe, count, and measure.

You can’t tell often in advance which observations will prove valuable. Do record them all! — Joseph Grinnell, 1908

Since we cannot interview the subject, we can only infer the past from the present. Ideally, a study should persist for at least the life span of an animal.

I organized behavioral codes to contain several levels of information. As in this example, if a child is outside playing with friends while minding her two-year-old sister, the activity was coded as 675: the 600 signifies noneconomic activity, the 70 that it is playing, and the 5 that it is playing while in charge of a child. All activities were coded in this way. A nested classificatory hierarchy preserves both detail for future research and flexibility to lump or disaggregate activities for analyses. This method of nesting information carries over into many kinds of coding and classificatory schemes.

Leave the drawing unfinished. Record as much information as you need, but don’t draw any forms, details, or colors that are merely repetitive. The back and front of a representative flower on a plant, for example, or half of a bilaterally symmetrical animal may be all that’s necessary.