The first and most important takeaway from Kashmir Hill’s excellent article in the New York Times about Mark, the man flagged by Google as a purveyor of Child Sexual Abuse Material (CSAM) for taking pictures of his son’s penis and sending them to their family doctor, and who subsequently lost nearly every aspect of his digital life when Google deleted his account, are the tremendous trade-offs entailed in the indiscriminate scanning of users’ cloud data.

On one hand, it seems like an incredible violation of privacy to have a private corporation effectively looking through every photo you upload, particularly when those uploads happen as part of the expected way in which your smartphone operates (users technically agree to this scanning, but as part of an endless End User License Agreement that is both ridiculously long and, more pertinently, inescapable if you want to use your phone as it was intended). Moreover, Google doesn’t simply scan for CSAM photos that are already known to exist via the PhotoDNA database of photos of exploited children; the company also leverages machine learning to look for new CSAM that hasn’t yet been identified as such.

On the other hand, as horrific as the material in the PhotoDNA database is, much of it has been floating around the Internet for years, which is to say the abuse depicted happened long ago; Google’s approach has the potential to discover abuse as it is happening, making it possible for the authorities to intercede and rescue the child in question. Hill’s story noted that in 2021 the CyberTipline at the National Center for Missing and Exploited Children, the only entity legally allowed to hold CSAM (NCMEC also manages the PhotoDNA database), “alerted authorities to ‘over 4,260 potential new child victims'”. We don’t know how many of those children were subsequently rescued, but a question worth posing to anyone unilaterally opposed to Google’s approach is how big that number would have to be to have made it worthwhile?

But, to return to the original hand, one of those 4,260 potential new child victims was Mark’s son (and another was taken by Cassio, a second father found by Hill caught in the same predicament, for the same reasons): the question for those applauding Google’s approach is how big the number of false positives would have to be to shut the whole thing down?

It was the exploration of these trade-offs that was at the heart of the Update I wrote about Hill’s story last week; as I noted there are no easy answers:

Nearly every aspect of this story is incredibly complex, and I understand and respect arguments on both sides: should there be scanning of cloud-related content? Should machine learning be leveraged to find new photos? Is it reasonable to obliterate someone’s digital life — except for what you give the police — given the possibility that they may be committing horrific crimes? These are incredibly difficult questions, particularly in the absence of data, because the trade-offs are so massive.

However, it seemed to me that one aspect of the case was very clear:

There is, though, one part of the story that is black-and-white. Google is unquestionably wrong to not restore the accounts in question. In fact, I am stunned by the company’s approach in these cases. Even if you grant the arguments that this awesome exercise of surveillance is warranted, given the trade-offs in question, that makes it all the more essential that the utmost care be taken in case the process gets it wrong. Google ought to be terrified it has this power, and be on the highest alert for false positives; instead the company has gone in the opposite direction, setting itself as judge, jury, and executioner, even when the people we have collectively entrusted to lock up criminals ascertain there was no crime. It is beyond arrogant, and gives me great unease about the company generally, and its long-term investments in AI in particular.

Not that it matters, one may argue: Google can do what they want, because they are a private company. That is an argument that may ring familar.

Tech and Liberty

In 2019 I discussed the distinction between public and private restrictions on speech in Tech and Liberty:

Alexander Hamilton was against the Bill of Rights, particularly the First Amendment. This famous xkcd comic explains why:

According to Randall Munroe, the author, the “Right to Free Speech” is granted by the First Amendment, which was precisely the outcome Hamilton feared in Federalist No. 84:

I go further, and affirm that bills of rights, in the sense and to the extent in which they are contended for, are not only unnecessary in the proposed Constitution, but would even be dangerous. They would contain various exceptions to powers not granted; and, on this very account, would afford a colorable pretext to claim more than were granted. For why declare that things shall not be done which there is no power to do? Why, for instance, should it be said that the liberty of the press shall not be restrained, when no power is given by which restrictions may be imposed? I will not contend that such a provision would confer a regulating power; but it is evident that it would furnish, to men disposed to usurp, a plausible pretense for claiming that power. They might urge with a semblance of reason, that the Constitution ought not to be charged with the absurdity of providing against the abuse of an authority which was not given, and that the provision against restraining the liberty of the press afforded a clear implication, that a power to prescribe proper regulations concerning it was intended to be vested in the national government. This may serve as a specimen of the numerous handles which would be given to the doctrine of constructive powers, by the indulgence of an injudicious zeal for bills of rights.

Hamilton’s argument is that because the U.S. Constitution was created not as a shield from tyrannical kings and princes, but rather by independent states, all essential liberties were secured by the preamble (emphasis original):

WE, THE PEOPLE of the United States, to secure the blessings of liberty to ourselves and our posterity, do ORDAIN and ESTABLISH this Constitution for the United States of America.

Hamilton added:

Here, in strictness, the people surrender nothing; and as they retain every thing they have no need of particular reservations.

Munroe, though, assumes the opposite: liberty, in this case the freedom of speech, is an artifact of law, only stretching as far as government action, and no further. Pat Kerr, who wrote a critique of this comic on Medium in 2016, argued that this was the exact wrong way to think about free speech:

Coherent definitions of free speech are actually rather hard to come by, but I would personally suggest that it’s something along the lines of “the ability to voluntarily express (and receive) opinions without suffering excessive penalties for doing so”. This is a liberal principle of tolerance towards others. It’s not an absolute, it isn’t comprehensive, it isn’t rigorously defined, and it isn’t a law.

What it is is a culture.

The context of that 2019 Article was the differing decisions between Facebook and Twitter in terms of allowing political ads on their platforms; over the ensuing three years the willingness and length to which these and other large tech platforms have been willing to go to police speech has expanded dramatically, even as the certainty that private censorship is ‘good actually’ has become conventional wisdom. I found this paragraph in a New York Times article about Elon Musk’s attempts to buy Twitter striking:

The plan jibes with Mr. Musk’s, Mr. Dorsey’s and Mr. Agrawal’s beliefs in unfettered free speech. Mr. Musk has criticized Twitter for moderating its platform too restrictively and has said more speech should be allowed. Mr. Dorsey, too, grappled with the decision to boot former President Donald J. Trump off the service last year, saying he did not “celebrate or feel pride” in the move. Mr. Agrawal has said that public conversation provides an inherent good for society. Their positions have increasingly become outliers in a global debate over free speech online, as more people have questioned whether too much free speech has enabled the spread of misinformation and divisive content.

In other words, the culture has changed; the law persists, but it does not and, according to the New York Times, ought not apply to private companies.

Scienter

The Google case is not about the First Amendment, either legally or culturally. The First Amendment is not absolute, and CSAM is an obvious example. In 1957’s Roth v. United States the Supreme Court held that obscene speech was not protected by the First Amendment; Justice William Brennan Jr. wrote:

All ideas having even the slightest redeeming social importance — unorthodox ideas, controversial ideas, even ideas hateful to the prevailing climate of opinion — have the full protection of the guaranties, unless excludable because they encroach upon the limited area of more important interests. But implicit in the history of the First Amendment is the rejection of obscenity as utterly without redeeming social importance. This rejection for that reason is mirrored in the universal judgment that obscenity should be restrained, reflected in the international agreement of over 50 nations, in the obscenity laws of all of the 48 States, and in the 20 obscenity laws enacted by the Congress from 1842 to 1956.

This reasoning is a reminder that laws ultimately stem from culture; still, the law being the law, definitions were needed, which the Supreme Court provided in 1973’s Miller v. California. Obscene works (1) appeal to the prurient interest in sex, (2) portrays in a patently offensive way sexual conduct specifically defined by a relevant law and (3) lack serious literary, artistic, political, or scientific value. The Supreme Court went further in terms of CSAM in 1982’s New York v. Ferber, holding that the harm inflicted on children is sufficient reason to make all forms of CSAM illegal, above and beyond the standards set forth by Miller. Justice Byron White wrote:

Recognizing and classifying child pornography as a category of material outside the protection of the First Amendment is not incompatible with our earlier decisions. “The question whether speech is, or is not, protected by the First Amendment often depends on the content of the speech”…

The test for child pornography is separate from the obscenity standard enunciated in Miller, but may be compared to it for the purpose of clarity. The Miller formulation is adjusted in the following respects: a trier of fact need not find that the material appeals to the prurient interest of the average person; it is not required that sexual conduct portrayed be done so in a patently offensive manner; and the material at issue need not be considered as a whole. We note that the distribution of descriptions or other depictions of sexual conduct, not otherwise obscene, which do not involve live performance or photographic or other visual reproduction of live performances, retains First Amendment protection. As with obscenity laws, criminal responsibility may not be imposed without some element of scienter on the part of the defendant.

“Scienter”, the “knowledge of the nature of one’s act”, is what ties this judicial history back to the original discussion of Google’s actions against Mark. As Hill explained in the New York Times:

I have seen the photos that Mark took of his son. The decision to flag them was understandable: They are explicit photos of a child’s genitalia. But the context matters: They were taken by a parent worried about a sick child.

The problem in this case comes from who is determining scienter.

Google and the Bill of Rights

Quite clearly Mark did not intend for the pictures he took for his son’s telemedicine to be used for pornographic purposes. The San Francisco Police Department, which had been notified by Google after a human reviewer confirmed the machine learning-driven discovery of Mark’s photos of his son, agreed. From Hill’s story:

In December 2021, Mark received a manila envelope in the mail from the San Francisco Police Department. It contained a letter informing him that he had been investigated as well as copies of the search warrants served on Google and his internet service provider. An investigator, whose contact information was provided, had asked for everything in Mark’s Google account: his internet searches, his location history, his messages and any document, photo and video he’d stored with the company.

The search, related to “child exploitation videos,” had taken place in February, within a week of his taking the photos of his son. Mark called the investigator, Nicholas Hillard, who said the case was closed. Mr. Hillard had tried to get in touch with Mark but his phone number and email address hadn’t worked. “I determined that the incident did not meet the elements of a crime and that no crime occurred,” Mr. Hillard wrote in his report. The police had access to all the information Google had on Mark and decided it did not constitute child abuse or exploitation.

Mark asked if Mr. Hillard could tell Google that he was innocent so he could get his account back. “You have to talk to Google,” Mr. Hillard said, according to Mark. “There’s nothing I can do.” Mark appealed his case to Google again, providing the police report, but to no avail…A Google spokeswoman said the company stands by its decisions, even though law enforcement cleared the two men.

In short, the questions about Google’s behavior are not about free speech; they do, though, touch on other Amendments in the Bill of Rights. For example:

• The Fourth Amendment bars “unreasonable searches and seizures”; while you can make the case that search warrants were justified once the photos in question were discovered, said photos were only discovered because Mark’s photo library was indiscriminately searched in the first place.
• The Fifth Amendment says no person shall be deprived of life, liberty, or property, without due process of law; Mark lost all of his data, email account, phone number, and everything else Google touched forever with no due process at all.
• The Sixth Amendment is about the rights to a trial; Mark was not accused of any crime in the real world, but when it came to his digital life Google was, as I noted, “judge, jury, and executioner” (the Seventh Amendment is, relatedly, about the right to a jury trial for all controversies exceeding $20).

Again, Google is not covered by the Bill of Rights; all of these Amendments, just like the First, only apply to the government. The reason why this case is useful, though, is it is a reminder that specific legal definitions are distinct from questions of right or wrong.

Working backwards, Google isn’t legally compelled to give Mark a hearing about his digital life (Sixth Amendment); they are wrong not to. Google isn’t legally compelled to give Mark due process before permanently deleting his digital life (Fifth Amendment); they are wrong not to. Google isn’t legally compelled to not search all of the photographs uploaded to Google (by default, if you click through all of the EULA’s); they are…well, this is where it gets complicated.

I started out this Article discussing the impossible trade-offs presented by questions of CSAM. People can and do make the case that to not search for this vileness, particularly if there is a chance that it can lead to the rescue of an abused child, is its own wrong. Resolving this trade-off in this way, though — that is, to violate the spirit and culture of the Fourth Amendment — makes it all the more essential to honor the spirit and culture of the Fifth and Sixth.

Paper Barriers

James Madison answered Hamilton’s objections in a speech to Congress introducing the Bill of Rights. What is interesting is that while Hamilton took it as a given that people would know and value their rights, Madison assumed the culture would run in the opposite direction, making an articulation of those rights important not just to restrain the government, but to remind the majority to not trample the rights of the minority:

But I confess that I do conceive, that in a Government modified like this of the United States, the great danger lies rather in the abuse of the community than in the Legislative body. The prescriptions in favor of liberty ought to be levelled against that quarter where the greatest danger lies, namely, that which possesses the highest prerogative of power. But this is not found in either the Executive or Legislative departments of Government, but in the body of the people, operating by the majority against the minority.

It may be thought that all paper barriers against the power of the community are too weak to be worthy of attention. I am sensible they are not so strong as to satisfy gentlemen of every description who have seen and examined thoroughly the texture of such a defence; yet, as they have a tendency to impress some degree of respect for them, to establish the public opinion in their favor, and rouse the attention of the whole community, it may be one means to control the majority from those acts to which they might be otherwise inclined.

This Article is a manifestation of Madison’s hope. Start with the reality that it seems quaint in retrospect to think that any of the Bill of Rights would be preserved absent the force of law. This is one of the great lessons of the Internet and the rise of Aggregators: when suppressing speech entailed physically disrupting printing presses or arresting pamphleteers, then restricting government, which retains a monopoly on real world violence, was sufficient to preserve speech. Along the same lines, there was no need to demand due process or a restriction on search and seizure on any entity but the government, because only the government could take your property or send you to jail.

Aggregators, though, make private action much more possible and powerful than ever before: yes, if you are kicked off of Twitter or Facebook, you can still say whatever you want on a street corner; similarly, if you lose all of your data and phone and email, you are still not in prison — and thank goodness that is the case! At the same time, it seems silly to argue that getting banned from a social media platform isn’t an infringement on individual free speech rights, even if it is the corporations’ own free speech rights that enable them to do just that legally, just as it is silly to argue that losing your entire digital life without recourse isn’t a loss of property without due process. The big Internet companies are manifesting Madison’s fears of the majority operating against the minority, and there is nothing the Bill of Rights can do about it.

What remains are those paper barriers, and what respect they might still engender, if it is possible to “rouse the attention of the whole community.” Rights are larger than laws, and Google has violated the former, even if they are not bound by the latter. The company ought not only change its policy with regards to Mark and Cassio, but fundamentally re-evaluate the balance it has struck between its unprecedented power over people’s lives and the processes it has in place to ensure that power is not abused. If it doesn’t, the people ought to, with what power they still conserve, do it for them.

I wrote a follow-up to this Article in this Daily Update.