This looks interesting. On the hand, it’s yet another proprietary creation by one browser vendor (boo!), but on the other hand it’s a declarative API with no JavaScript required (yay!).
Even if this particular feature doesn’t work out, I hope that this is the start of a trend for declarative access to browser features.
A few months back, I wrote about how Google is breaking its social contract with the web, harvesting our content not in order to send search traffic to relevant results, but to feed a large language model that will spew auto-completed sentences instead.
I still think Chris put it best:
I just think it’s fuckin’ rude.
When it comes to the crawlers that are ingesting our words to feed large language models, Neil Clarke describes the situtation:
It should be strictly opt-in. No one should be required to provide their work for free to any person or organization. The online community is under no responsibility to help them create their products. Some will declare that I am “Anti-AI” for saying such things, but that would be a misrepresentation. I am not declaring that these systems should be torn down, simply that their developers aren’t entitled to our work. They can still build those systems with purchased or donated data.
Alas, the current situation is opt-out. The onus is on us to update our robots.txt file.
Neil handily provides the current list to add to your file. Pass it on:
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: FacebookBot
Disallow: /
In theory you should be able to group those user agents together, but citation needed on whether that’s honoured everywhere:
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Google-Extended
User-agent: Omgilibot
User-agent: FacebookBot
Disallow: /
There’s a bigger issue with robots.txt though. It too is a social contract. And as we’ve seen, when it comes to large language models, social contracts are being ripped up by the companies looking to feed their beasts.
As Jim says:
I realized why I hadn’t yet added any rules to my
robots.txt: I have zero faith in it.
That realisation was prompted in part by Manuel Moreale’s experiment with blocking crawlers:
So, what’s the takeaway here? I guess that the vast majority of crawlers don’t give a shit about your
robots.txt.
Time to up the ante. Neil’s post offers an option if you’re running Apache. Either in .htaccess or in a .conf file, you can block user agents using mod_rewrite:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (CCBot|ChatGPT|GPTBot|Omgilibot| FacebookBot) [NC]
RewriteRule ^ – [F]
You’ll see that Google-Extended isn’t that list. It isn’t a crawler. Rather it’s the permissions model that Google have implemented for using your site’s content to train large language models: unless you opt out via robots.txt, it’s assumed that you’re totally fine with your content being used to feed their stochastic parrots.
I really hope that Betteridge’s Law doesn’t apply to this headline.
While the dream of “personalized” ads has turned out to be mostly a nightmare, adtech has built some of the wealthiest companies in the world based on tracking us. It’s no surprise to me that as Members of the European Parliament contemplate tackling these many harms, Big Tech is throwing millions of Euros behind a “necessary evil” PR defense for its business model.
But tracking is an unnecessary evil.
Even in today’s tracking-obsessed digital ecosystem it’s perfectly possible to target ads successfully without placing people under surveillance. In fact right now, some of the most effective and highly valued online advertising is contextual — based on search terms, other non-tracking based data, and the context of websites rather than intrusive, dangerous surveillance.
Let’s be clear. Advertising is essential for small and medium size businesses, but tracking is not.
Rather than creating advertising that is more relevant, more timely and more likable we are creating advertising that is more annoying, more disliked, and more avoided.
I promise you, the minute tracking is outlawed, Facebook, Google and the rest of the adtech giants will claim that their new targeting mechanisms (whatever they turn out to be) are superior to tracking.
Behavioral ads are only more profitable than context ads if all the costs of surveillance – the emotional burden of being watched; the risk of breach, identity-theft and fraud; the potential for government seizure of surveillance data – is pushed onto internet users. If companies have to bear those costs, behavioral ads are a total failure, because no one in the history of the human race would actually grant consent to all the things that gets done with our data.
Google and the entire tracking industry relies on IAB Europe’s consent system, which has now been found to be illegal.
There’s an interesting thread on Github about the tongue-twistingly named beforeinstallpromt JavaScript event.
Let me back up…
Progressive web apps. You know what they are, right? They’re websites that have taken their vitamins. Specifically, they’re responsive websites that:
The web app manifest—a JSON file of metadata—is particularly useful for describing how your site should behave if someone adds it to their home screen. You can specify what icon should be used. You can specify whether the site should launch in a browser or as a standalone app (practically indistinguishable from a native app). You can specify which URL on the site should be used as the starting point when the site is launched from the home screen.
So progressive web apps work just fine when you visit them in a browser, but they really shine when you add them to your home screen. It seems like pretty much everyone is in agreement that adding a progressive web app to your home screen shouldn’t be an onerous task. But how does the browser let the user know that it might be a good idea to “install” the web site they’re looking at?
The Samsung Internet browser does ambient badging—a + symbol shows up to indicate that a website can be installed. This is a great approach!
I hope that Chrome on Android will also use ambient badging at some point. To start with though, Chrome notified users that a site was installable by popping up a notification at the bottom of the screen. I think these might be called “toasts”.
Needless to say, the toast notification wasn’t very effective. That’s because we web designers and developers have spent years teaching people to immediately dismiss those notifications without even reading them. Accept our cookies! Sign up to our newsletter! Install our native app! Just about anything that’s user-hostile gets put in a notification (either a toast or an overlay) and shoved straight in the user’s face before they’ve even had time to start reading the content they came for in the first place. Users will then either:
X in the corner of the notification.A tiny fraction of users might actually click on the call to action, possibly by mistake.
Chrome didn’t abandon the toast notification for progressive web apps, but it did change when they would appear. Rather than the browser deciding when to show the prompt—usually when the user has just arrived on the site—a new JavaScript event called beforeinstallprompt can be used.
It’s a bit weird though. You have to “capture” the event that fires when the prompt would have normally been shown, subdue it, hold on to that event, and then re-release it when you think it should be shown (like when the user has completed a transaction, for example, and having your site on the home screen would genuinely be useful). That’s a lot of hoops. Here’s the code I use on The Session to only show the installation prompt to users who are logged in.
The end result is that the user is still shown a toast notification, but at least this time it’s the site owner who has decided when it will be shown. The Chrome team call this notification “the mini-info bar”, and Pete acknowledges that it’s not ideal:
The mini-infobar is an interim experience for Chrome on Android as we work towards creating a consistent experience across all platforms that includes an install button into the omnibox.
I think “an install button in the omnibox” means ambient badging in the browser interface, which would be great!
Anyway, back to that thread on Github. Basically, neither Apple nor Mozilla are going to implement the beforeinstallprompt event (well, technically Mozilla have implemented it but they’re not going to ship it). That’s fair enough. It’s an interim solution that’s not ideal for all reasons I’ve already covered.
But there’s a lot of pushback. Even if the details of beforeinstallprompt are troublesome, surely there should be some way for site owners to let users know that can—or should—install a progressive web app? As a site owner, I have a lot of sympathy for that viewpoint. But I also understand the security and usability issues that can arise from bad actors abusing this mechanism.
Still, I have to hand it to Chrome: even if we put the beforeinstallprompt event to one side, the browser still has a mechanism for letting users know that a progressive web app can be installed—the mini info bar. It’s not a great mechanism, but it’s better than nothing. Nothing is precisely what Firefox and Safari currently offer (though Firefox is experimenting with something).
In the case of Safari, not only do they not provide a mechanism for letting the user know that a site can be installed, but since the last iOS update, they’ve buried the “add to home screen” option even deeper in the “sharing sheet” (the list of options that comes up when you press the incomprehensible rectangle-with-arrow-emerging-from-it icon). You now have to scroll below the fold just to find the “add to home screen” option.
So while I totally get the misgivings about beforeinstallprompt, I feel that a constructive alternative wouldn’t go amiss.
And that’s all I have to say about that.
Except… there’s another interesting angle to that Github thread. There’s talk of allowing sites that are launched from the home screen to have access to more features than a site inside a web browser. Usually permissions on the web are explicitly granted or denied on a case-by-case basis: geolocation; notifications; camera access, etc. I think this is the first time I’ve heard of one action—adding to the home screen—being used as a proxy for implicitly granting more access. Very interesting. Although that idea seems to be roundly rejected here:
A key argument for using installation in this manner is that some APIs are simply so powerful that the drive-by web should not be able to ask for them. However, this document takes the position that installation alone as a restriction is undesirable.
I understand that Chromium or Google may hold such a position but Apple’s WebKit team may not necessarily agree with such a position.
Yesterday I wrote about how much I’d like to see silent push for the web:
I’d really like silent push for the web—the ability to update a cache with fresh content as soon as it’s published; that would be nifty! At the same time, I understand the concerns. It feels more powerful than other permission-based APIs like notifications.
Today, John Holt Ripley responded on Twitter:
hi there, just read your blog post about Silent Push for acthe web, and wondering if Periodic Background Sync would cover a few of those use cases?
Periodic background sync looks very interesting indeed!
It’s not the same as silent push. As the name suggests, this is about your service worker waking up periodically and potentially fetching (and caching) fresh content from the network. So the service worker is polling rather than receiving a push. But I’ll take it! It’s definitely close enough for the kind of use-cases I’ve been thinking about.
Interestingly, periodic background sync also ties into the other part of what I was writing about: permissions. I mentioned that adding a site the home screen could be interpreted as a signal to potentially allow more permissions (or at least allow prompts for more permissions).
Well, Chromium has a document outlining metrics for attempting to gauge site engagement. There’s some good thinking in there.
After Indie Web Camp in Berlin last year, I wrote about Seb’s nifty demo of push without notifications:
While I’m very unwilling to grant permission to be interrupted by intrusive notifications, I’d be more than willing to grant permission to allow a website to silently cache timely content in the background. It would be a more calm technology.
Phil Nash left a comment on the Medium copy of my post explaining that Seb’s demo of using the Push API without showing a notification wouldn’t work for long:
The browsers allow a certain number of mistakes(?) before they start to show a generic notification to say that your site sent a push notification without showing a notification. I believe that after ~10 or so notifications, and that’s different between browsers, they run out of patience.
He also provided me with the name to describe what I’m after:
You’re looking for “silent push” as are many others.
Silent push is something that is possible in native apps. It isn’t (yet?) available on the web, presumably because of security concerns.
It’s an API that would ripe for abuse. I mean, just look at the mess we’ve made with APIs like notifications and geolocation. Sure, they require explicit user opt-in, but these opt-ins are seen so often that users are sick of seeing them. Silent push would be one more permission-based API to add to the stack of annoyances.
Still, I’d really like silent push for the web—the ability to update a cache with fresh content as soon as it’s published; that would be nifty! At the same time, I understand the concerns. It feels more powerful than other permission-based APIs like notifications.
Maybe there could be another layer of permissions. What if adding a site to your home screen was the first step? If a site is running on HTTPS, has a service worker, has a web app manifest, and has been added to the homescreen, maybe then and only then should it be allowed to prompt for permission to do silent push.
In other words, what if certain very powerful APIs were only available to progressive web apps that have successfully been added to the home screen?
Frankly, I’d be happy if the same permissions model applied to web notifications too, but I guess that ship has sailed.
Anyway, all this is pure conjecture on my part. As far as I know, silent push isn’t on the roadmap for any of the browser vendors right now. That’s fair enough. Although it does annoy me that native apps have this capability that web sites don’t.
It used to be that there was a long list of features that only native apps could do, but that list has grown shorter and shorter. The web’s hare is catching up to native’s tortoise.
Now this is a feature request I can get behind!
A user must provide permission to enable geolocation, or notifications, or camera access, so why not also require permission for megabytes of JavaScript that will block the main thread?
Without limits, there is no incentive for a JavaScript developer to keep their codebase small and dependencies minimal. It’s easy to add another framework, and that framework adds another framework, and the next thing you know you’re loading tens of megabytes of data just to display a couple hundred kilobytes of content.
I’m serious about this. It’s is an excellent proposal for WebKit, similar to the never-slow mode proposed by Alex for Chromium.
Remember when I wrote about using push without notifications? Sebastiaan has written up the details of the experiment he conducted at Indie Web Camp Berlin.
On the first day of Indie Web Camp Berlin, I led a session on going offline with service workers. This covered all the usual use-cases: pre-caching; custom offline pages; saving pages for offline reading.
But on the second day, Sebastiaan spent a fair bit of time investigating a more complex use of service workers with the Push API.
The Push API is what makes push notifications possible on the web. There are a lot of moving parts—browser, server, service worker—and, frankly, it’s way over my head. But I’m familiar with the general gist of how it works. Here’s a typical flow:
Here’s what Sebastiaan wanted to investigate: what if that last step weren’t so intrusive? Here’s the alternate flow he wanted to test:
It worked.
I think this could be a real game-changer. I don’t know about you, but I’m very, very wary of granting websites the ability to send me push notifications. In fact, I don’t think I’ve ever given a website permission to interrupt me with push notifications.
You’ve seen the annoying permission dialogues, right?
In Firefox, it looks like this:
Will you allow name-of-website to send notifications?
[Not Now] [Allow Notifications]
In Chrome, it’s:
name-of-website wants to
Show notifications
[Block] [Allow]
But in actual fact, these dialogues are asking for permission to do two things:
There’s no way to ask for permission just to do the first part. That’s a shame. While I’m very unwilling to grant permission to be interrupted by intrusive notifications, I’d be more than willing to grant permission to allow a website to silently cache timely content in the background. It would be a more calm technology.
Think of the use cases:
Then when I’m on a plane, or in the subway, or in any other situation without a network connection, I could still visit these websites and get content that’s fresh to me. It’s kind of like background sync in reverse.
There’s plenty of opportunity for abuse—the cache could get filled with content. But websites can already do that, and they don’t need to be granted any permissions to do so; just by visiting a website, it can add multiple files to a cache.
So it seems that the reason for the permissions dialogue is all about displaying notifications …not so much about receiving push messages from the server.
I wish there were a way to implement this background-caching pattern without requiring the user to grant permission to a dialogue that contains the word “notification.”
I wonder if the act of adding a site to the home screen could implicitly grant permission to allow use of the Push API without notifications?
In the meantime, the proposal for periodic synchronisation (using background sync) could achieve similar results, but in a less elegant way; periodically polling for new content instead of receiving a push message when new content is published. Also, it requires permission. But at least in this case, the permission dialogue should be more specific, and wouldn’t include the word “notification” anywhere.
Sally takes a long hard look at permissions on the web. It’s a fascinating topic because of all the parties involved—browsers, developers, and users.
In order to do permissions well, I think there are two key areas to think about - what’s actually being requested, and how it’s being requested.
Is a site being intrusive with what they can potentially learn about me (say, wanting my precise location when it’s unnecessary)? Or is it being intrusive in terms of how they interact with me (popping up a lot of notifications and preventing me from quickly completing my intended task)? If one of those angles doesn’t work well, then regardless of whether the other is acceptable to someone, they’re likely to start opting out and harbouring negative feelings.
A collection of interface patterns for granting or denying permissions.
Software is politics, because software is power.
The transcript of a tremendous talk by Richard Pope.
This a great proposal: well-researched and explained, it tackles the tricky subject of balancing security and access to native APIs.
Far too many ideas around installable websites focus on imitating native behaviour in a cargo-cult kind of way, whereas this acknowledges addressability (with URLs) as a killer feature of the web …a beautiful baby that we definitely don’t want to throw out with the bathwater.
A one-stop-shop with links to the authentication settings of various online services. Take the time to do a little Spring cleaning.