oh wow, hey @csarven. apologies, i totally missed this. absolutely, will do.

done! your site is now gone from the dataset. details.

fwiw, here's what i see in http://csarven.ca/robots.txt right now. i do get that you may want to allow some crawlers but not others, like indie map.

User-agent: *
Disallow: /archives
Disallow: /scripts
Disallow: /url
Disallow: /labs
Disallow: /presentations
Disallow: /webstream
Disallow: /search
Disallow: /statistical-linked-dataspaces-and-analysis-overview
Allow: /labs/indexability
Allow: /archives/articles

Thanks!

Pardon me if I'm looking at the wrong code, but perhaps it is also worthwhile to update the user-agent value: https://github.com/snarfed/indie-map/blob/master/crawl/wget.sh#L9 ?

interesting idea. you mean, set it to Indie Map? i could! technically the user agent still is wget though, right? Indie Map is just the use case? maybe I'm splitting hairs.

I would classify indie-map (the software of this repository) as the user-agent, as opposed to a particular library that's doing the fetching. Just as Firefox uses its own library to negotiate resources. But yes, generally it can be arbitrary.

So, you can do something like User-Agent: indie-map or if you're feeling adventurous use User-Agent: https://github.com/snarfed/indie-map/ or some other HTTP URI that can provide a structured description for the application eg view-source https://dokie.li/

also fwiw, i've only actually done this whole crawl once, and i have no plans to do it again right now, regularly or otherwise.

still, good idea. thanks for the nudge.