The Magic Cookie: How Lou Montulli cured the Web’s amnesia

BY STEVEN JOHNSON

More than two decades ago, Web pioneer Lou Montulli invented browser software cookies with the goal of preserving user privacy online. But then things took an unexpected turn.

Illustration: Karol Banach

Lou Montulli

What is the smallest form that a genuinely world-changing idea can take? If you had to organize the truly disruptive ideas by how well they compressed as data, what would that ranking look like? No doubt, the canonical equations of physics and geometry and algebra would be at the top of the list. But somewhere close behind would be this unlikely sequence of text that a 23-year-old programmer named Lou Montulli wrote, sometime in the fall of 1994:

Courtesy of Lou Montulli

The text was the opening salvo of a short standards document that Montulli was working on as one of the first employees at the pathbreaking dot-com startup Netscape. Montulli was already a pioneer in those early days of the Web; he’d programmed one of the first hypertext browsers, Lynx, and taken the lead on several important additions to the HTML standard. But this little snippet of code—its file size limited to 4K—would go on to have a momentous impact on the world, one that in many ways ran counter to Montulli’s original motivation for writing it. Variations of that code are almost certainly installed on the computer you are using to read this essay, and billions of dollars in advertising revenue depend on it.

Lou Montulli was inventing with that sequence a new way of sharing identity on the Web, one that would become arguably the most important—and ultimately controversial—new standard introduced since the very early years of HTML: the cookie.

Invention of a Web

Born in 1970 to a military family, Montulli shuffled from army base to army base as a child, before settling in Kansas for his teenage years. Personal computers were just becoming a reality in those days, but they were not an early passion for Montulli.

Montulli in the early 90s. Courtesy of Lou Montulli

“I took a computer science course in high school,” he recalls now with a smile, “which I mostly took because I heard that the teacher didn't really like to do anything except let you play video games. It wasn't because I was obsessed with computers—it just sounded like more fun than some of the other things that I could do in high school. But I did learn a little bit of Pascal programming in there, which made me think, okay, I can program.”

A few years later, he found himself at the University Of Kansas, working a side job doing tech support and other odds and ends for the university’s computing center. “We had this project at the university that had been kind of back-burnered for a while,” he says, “where we wanted to start what we called the ‘campus wide information system,’ which in today's world we would call a website.”

Unwittingly, Montulli had stumbled into a fascinating—and mostly forgotten—transitional period in digital communications. In the late 80s and early 90s, a critical mass of network-connected computers—some of them PCs attached to dial-up modems, some of them institutional mainframes—had emerged, particularly on university campuses. What had seemed like the stuff of science fiction just a few years before—ordinary people sitting at their home computers pulling down information from other computers around the world—now seemed within reach. But there was no standardized way of exploring that information space. A university could put some documents online, and you could use applications like Gopher to see a list of those documents and retrieve the ones that looked promising. But there was no way to search the documents, no way to connect them. It was like exploring a library exclusively by walking through the stacks, scanning the titles on each book’s spine.

The early days of online surfing

It’s important to remember just how fragmented the network experience was in the early 90s, before the mass adoption of the Web. “Going online” could mean anything from visiting a small, dial-up bulletin board service maintained by a private group or a school, to using a proprietary commercial network like Compuserve or Delphi, to using an Internet-connected computer—most likely at a university or large corporation—where you could send email to other users much as we do today. Because the digital world had not settled on a common standard for sharing pages of information, interacting with many of these services was like living on a remote island with its own limited population, cut off from the rest of the global village.

Hyperlinked library. Illustration: Karol Banach

Montulli had no way of knowing it at the time, but he was about to embark on an exceptionally productive period where he would help develop some of the core elements that ultimately defined the way people represent themselves—sharing ideas and identity—online. At some point in 1991, Montulli found a demo of a new application called HyperRez, which used a different model for exploring information space, one that computer scientists and experimental fiction writers had been experimenting with for the previous decade or two: hypertext. Instead of browsing static files, pulling them down one at a time, and reading through them in a linear fashion, hypertext let you jump from page to page, following hyperlinked words.

Before long, Montulli began tinkering with the idea of connecting the Gopher file system with a hypertext frontend. “I figured, well, how hard could it be to just glue the two together?” Looking back on the project, he chuckles at how little he understood about the underlying code at the time. “I have no idea how I actually did it,” he says. “I think one of my one of my skills is that I'm able to just understand things just enough to get them to work together.”

“I figured, well, how hard could it be to just glue the two together?”

HyperRez

HyperRez was one of the early precursors of modern web browsers. Developed by Neil Larson, this DOS file browser program was built on the capabilities of Houdini, a knowledge network program that supported 2,500 topics cross-connected with 7,500 links in each file along with hypertext links. HyperRez’s hypertext engine played a crucial role in the creation of Lynx which gave momentum to the invention of the Web.

The program Montulli hacked together with a few other students turned out to have wider utility beyond just the “campus-wide information system” at the University of Kansas. Other universities adopted it to share their own data archives. Eventually, he gave it a name: Lynx. It was one of the world’s first hypertext browsers, though in its early days it wasn’t connected to the World Wide Web, which had just been officially announced by Tim Berners-Lee at CERN in Geneva. As Michael Grober, one of the Lynx co-creators, later recalled, “I like to say that we invented a Web. Rather than the Web, of course.”

“The momentum behind hypertext network hypertext was there—it was in the air,” Montulli says. “I didn't know it, but Tim Berners-Lee had already started working on HTML before I wrote the first version of Lynx. It's just that nobody knew about it because it was stuck in a little corner of Europe.”

Lynx: The oldest web browser

Lynx turned 30 in 2022, making it the oldest web browser still being maintained. Currently, a group of volunteers led by Thomas Dickey make sure the latest version is up to date.

“I think one of my skills is that I'm able to just understand things just enough to get them to work together.”

When HTML started to pick up steam, Montulli adapted Lynx to be able to read files in that format. Not long after that, Marc Andreessen launched Mosaic out of the University of Illinois in Urbana–Champaign. (Most people associate the Web revolution with Silicon Valley, but in many ways its true geographic roots were Geneva and the American Midwest.) Montulli found himself a central participant in an international conversation, happening through USENET, wrestling with the technical specs for a genuinely new medium. “It was the Wild West for releasing stuff and trying to make it all work together. It's amazing that we were able to have such agreement, but none of us had a lot of ego about things and nor did we have any corporate vested interest in a particular direction,” he says now. “By the time it got to 1993, the small group of people who were working on this were really convinced that we were building the information superhighway that Al Gore was talking about. We wanted the Web to be that thing.”

Web revolution beyond Silicon Valley

While the dot-com boom of the late 90s and the social media revolution would all be driven by tech juggernauts based on the West Coast—mostly in Seattle and the Bay Area—the initial roots of the online revolution were more diverse geographically: Berners-Lee devising HTML and HTTP at CERN in Geneva, Andreesen writing Mosaic in Illinois, Montulli building Lynx in Kansas. Even the then-giant proprietary network AOL was based in northern Virginia. Some of the tech titans on the West Coast underestimated the significance of the coming Web revolution. In his 1995 book on the future of technology, The Road Ahead, Bill Gates famously only mentioned the Internet a few times—though to his credit, he soon recognized his oversight and made the web browser Internet Explorer a major priority for Microsoft in the next few years.

“We were really convinced that we were building the information superhighway that Al Gore was talking about. We wanted the Web to be that thing.”

“Who are you again?”

By 1994, Montulli had moved to the Bay Area as one of the founding employees of Netscape, the browser company that would go on to become a flagship firm of the dot-com boom. His tenure there is now most famous for his work on cookies, but Montulli made a number of other important contributions to the Web that have since become second nature to us. Perhaps the most important was the introduction of forms, a feature that Montulli had already developed at Lynx. Every time you type a search query into Google or type up a comment to post on a discussion board, you are using the descendants of the HTML forms that Montulli first sketched out in 1992. Even in its early days, the Web had been heralded as a new model of “interactive” media, but the truth of the matter is that the first iteration of the Web that Berners-Lee introduced wasn’t really a two-way medium. It was entirely static pages, connected by links. As a user, you could “interact” with the medium by choosing to click on one link or another. That was the only feedback mechanism. Forms made it possible for end-users to actually contribute information directly from within a web browser. They were a crucial early step in turning the Web into a social medium, a virtual space where different people could share ideas on a single page.

Netscape

The private company Netscape was the brainchild of Andreesen and veteran Silicon Valley executive Jim Clark, who had founded the pioneering graphics workstation company Silicon Graphics more than a decade earlier. After launching a browser similar to Andreesen’s original Mosaic—but based on completely different code to avoid any intellectual property conflicts with the University of Illinois—Netscape went public in 1995 with a spectacular first day of trading that saw its share price more than triple. Netscape’s IPO—followed shortly by Yahoo’s—launched a five-year period of frenzied activity in the Internet space now known as the dot-com boom. Netscape was later acquired by AOL in 1998, and officially disbanded in 2003.

Interface of the Netscape browser

“It’s a funny story because when I proposed adding forms, Tim Berners-Lee came back and said, ‘No, no, we don't want to do that,’” Montulli recalls. “He thought everything should be done with links.”

There was another fundamental limitation to the Web back in those days: it had a very short attention span. The Web had been designed for a certain kind of efficiency: a user would connect to a web server, grab a document, and then disconnect, freeing up the server to share another document with another user. There was no concept of a “user session” where the server could recall your preferences or your identity as you moved from page to page. “[It was] a bit like talking to someone with Alzheimer's disease,” Montulli would later write. “Each interaction would result in having to introduce yourself again, and again, and again.”

“It was a bit like talking to someone with Alzheimer's disease. Each interaction would result in having to introduce yourself again, and again, and again.”

Web’s amnesia. Illustration: Karol Banach

The easiest way around this limitation would have been to give each web browser some kind of unique identifier that they would use while exploring the Web, but to Montulli and his colleagues, that seemed too problematic in terms of user privacy. With one single ID, sites would effectively be able to compare notes to track your browsing history around the Web.

This was the problem that Montulli found himself wrestling with in the middle of 1994. He was solving a very local problem: how do you create a sustained user session without compromising the user’s privacy. The solution he came up with had an elegant simplicity to it. It accomplished just about everything the Netscape team had wanted.

But then, as it happened, Montulli’s idea turned out to be a key ingredient in another innovation that would emerge a few years later: surveillance capitalism.

Surveillance capitalism

“Surveillance capitalism” is a concept popularized by Harvard professor Shoshana Zuboff—most famously in her 2018 book, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. Zuboff argued that the rise of companies like Google and Facebook, who profited by collecting and analyzing information about their users, represented a new model of capitalism equivalent to the mass production model defined by companies like Ford and General Motors a century before. “In an information civilization,” Zuboff writes, “societies are defined by questions of knowledge—how it is distributed, the authority that governs its distribution and the power that protects that authority. Who knows? Who decides who knows? Who decides who decides who knows? Surveillance capitalists now hold the answers to each question, though we never elected them to govern.”

A software trick that cured the Web’s amnesia

“So, yeah, the cookie,” Montulli says with a laugh. “It's one week of my life that turned into the most important thing that I ever did.”
While the Web’s general amnesia had always been a known limitation of the platform, commercial interests had finally compelled the Netscape team to find a way around the problem. Amazon was still several years away from being founded, but already there was talk of using this nascent medium as a platform for commerce.
“The specific use case that we were discussing at the time was to have a shopping cart model,” Montulli says. “So you're browsing items, and you say: I like those shoes. You click on the button that says: I want to buy these things. And you expect it to go in the cart—and not just disappear.”
At some point in the summer of 1994, while mulling different ways to create a sustained user session—without giving up too much privacy—Montulli remembered a software trick from an old operating systems manual he’d read a few years earlier, a technique for passing information back and forth between the user and the system. For some reason, the small piece of data exchanged had been called a “magic cookie.”

“It's one week of my life that turned into the most important thing that I ever did.”

Inspired by that earlier model, Montulli sketched out an architecture for a web-based “cookie” that would give the medium a sense of memory without compromising privacy. In Montulli’s system, the basic act of requesting a web page from a server would be accompanied by a small but significant additional step: when the page information was sent back to your browser, it would be accompanied by a tiny file—no more than 4K in size—that contained unique information that would identify you as a user. That information would be accessible only to you and the specific web server you were interacting with. The cookie served to you by the Yahoo web server would only be visible to Yahoo. If you went to another site—say Wired.com—they could create their own cookie to give you a persistent identity on their site, but they’d have no way of detecting the existence of the Yahoo cookie.

A magic cookie. Illustration: Karol Banach

For the first time—thanks to Montulli’s cookie—the Web was able to have something like a fluid memory for individuals; you could move through a site and have your actions captured without having to sign in with your password every time you reloaded a page. Today, the entire experience of the Web is shaped by the invisible presence of cookies: when you pull up a browser window and automatically log onto Twitter or The New York Times; when you type information over a sequence of pages to book an airline ticket; and yes, when you put those new sneakers in your shopping cart and find that they’re still there waiting for checkout two days later. All that was made possible by a few lines of code that Montulli repurposed from an old programming manual.
“Almost all of us in the world who are innovating are simply taking ideas that were from some other adjacent space and adapting them and pulling them into our space,” Montulli says now. “It's rare that the idea pops whole cloth into somebody's mind.”

“Almost all of us in the world who are innovating are simply taking ideas that were from some other adjacent space and adapting them and pulling them into our space.”

Accidental rise of a billion-dollar advertising model

Like most important technological breakthroughs, the cookie turned out to trigger a few long-term consequences that ran against the original aims of its inventor. While Montulli’s solution created an elegant “sandbox” that preserved a one-to-one relationship between the user and the website they were visiting, it had one critical vulnerability, one that arose not from the architecture of the cookies so much as from the way the Web handled image files. When you request a page from a web server, it sends the HTML file associated with that address to your browser (along with a cookie, if needed.) But it can also send *references* to image files that reside on other web servers, so the final page you assemble might have text from a local newspaper’s web server, but images served from Wikipedia’s servers. All this happens transparently from the end user’s perspective: it seems as though you’re just interacting with the local paper, but in fact, behind the scenes, your browser is pulling down information from two unrelated servers.

Within a year of Montulli’s cookie framework becoming accepted by the web community, a handful of pioneering online ad agencies—led by a new New York-based startup called DoubleClick, now part of Google—realized that they could use those image files to do exactly the kind of user tracking that Montulli had been trying to prevent. When a user visited a local newspaper site, she might receive a cookie from the news server that would give her a persistent session browsing the paper, but she could also receive a cookie from a separate server that supplied an ad banner, or even worse, an invisible graphic inserted on the page purely for tracking purposes. If all of those images—the ad banner, the invisible graphics—pointed back to the same server, it was suddenly possible to track individual users as they moved around the Web, so long as they were visiting sites that relied on the same ad server.

A path to third-party cookies

The New York-based DoubleClick was one of the first businesses to prove that the advertising model could work on the Web, thanks in large part to the way it hacked Montulli’s cookie standard to allow user-tracking across multiple websites. After going public in 1998, the company was acquired by Google, and some of its ad-serving tools were integrated into Google’s massively profitable AdWords system. Over time, the third-party cookie technique pioneered by DoubleClick would be adopted throughout the online advertising market.

Initially, the ad companies used this technique to track unique users to ensure they weren’t double-counting the overall audience that viewed a given ad. But eventually, “third-party cookies,” as they came to be called, enabled the kind of deep—some would say sinister—ad targeting that would become the dominant business model of search, news, and social media. Today, when you visit that local newspaper and see ads for Caribbean vacations a few days after searching Google for info on a resort in the Bahamas—it’s third-party cookies that make that targeted ad possible.

“Third-party cookies,” as they came to be called, enabled the kind of deep—some would say sinister—ad targeting that would become the dominant business model of search, news, and social media.

Surveillance capitalism. Illustration: Karol Banach

Montulli recalls being shocked when he first encountered the DoubleClick implementation in the late 90s. “My first thought was: ‘Holy shit, that shouldn't be possible—we tried to design cookies so that would not be possible,’” he says. “After I dove in and figured out how DoubleClick was doing it, we started talking about the ways in which we could respond. We ended up taking the path of creating a set of tools to allow users to see and control the ways in which cookies can be used.”

“We tried to design cookies so that would not be possible.”

“Cookies by themselves are not a bad actor—but cookies plus images served from third parties all work together to allow ad trackers,” Montulli points out. “This is the most important caveat: You have to have a bunch of websites, all essentially conspiring together to allow this to happen. You can't be a black hat hacker stealing cookie data—because you have to be on hundreds of websites that all have to point to a big public URL. So cookies are quite secure from that perspective.” The cookie didn’t create a traditional software vulnerability where an individual rogue hacker could get access to your private data. Instead, it created a vulnerability—thanks to those third-party images—where giant corporations could get access to your private data.

“Cookies by themselves are not a bad actor.”

The law of unintended consequences

In time, that minuscule snippet of code that Lou Montulli wrote in 1994 would inspire major pieces of international regulation, like the EU’s 2009 ePrivacy Directive, which compelled web providers to receive consent from users before installing cookies on their machines, or Apple’s recent restrictions on third-party cookies, which has dealt a significant blow to Facebook’s advertising business and contributed to a $100 billion reduction in their market cap. Many people believe that both the Trump Presidential campaign and Brexit would have failed in the 2016 polls had targeted advertising, made possible by third-party cookies, not had such a powerful effect on popular opinion.

Yet at the same time, the cookie—even the third-party variations—has enabled countless online developments that are clearly positive in nature. A Web without persistent user sessions would have been a greatly diminished medium, and an enormously frustrating one. And perhaps with these latest regulations and privacy features, we are slowly finding our way back to the original values that motivated Montulli to invent the cookie in the first place: somewhere in the Goldilocks zone between an amnesiac Web and an Orwellian one.

”I think the law of unintended consequences definitely applies here. I do believe people should think long and hard about the consequences of what they are building. But I also think it’s not possible to be imaginative enough to come up with all of the possibilities.”

Butterfly effect. Illustration: Karol Banach

I asked Montulli if he thought software creators could be more mindful of the downstream implications of their innovations—whether they could better anticipate the ways their ideas might be manipulated once they were released into the world. “That is a really interesting question,” he told me. “I think the law of unintended consequences definitely applies here. I do believe people should think long and hard about the consequences of what they are building. But I also think it’s not possible to be imaginative enough to come up with all of the possibilities.”
Years ago, the mathematician Edward Lorenz proposed a metaphor to describe how very small elements in a system’s initial conditions can lead to momentous changes over time. Imagining a tornado that ultimately emerges out of the tiny air perturbations caused by the flapping of a butterfly’s wings, Lorenz called it the “butterfly effect.” For better and for worse, Montulli’s cookie may be the most pronounced example of a technological butterfly effect in our time. But instead of a butterfly flapping its wings, it’s a 23-year-old programmer writing a few lines of code to make a shopping cart feature work. Almost three decades later, we’re still riding out the storm that code helped create.

Steven Johnson is the bestselling author of 13 books, including Where Ideas Come From. He’s the host of the PBS/BBC series Extra Life and How We Got to Now. He regularly contributes to The New York Times Magazine and has written for Wired, The Guardian, and The Wall Street Journal. His TED Talks on the history of innovation have been viewed more than ten million times.