Application and specification to forward and/or move data between Solid Pods

While we were working on solid-nextcloud, a community solid server shut down. This server was well-used and contained a lot of datapods. While there was time to copy the data and store it somewhere else, it did a lot of damage still. All links to these satapods broke and had to be fixed manually. Not every link could be altered, for example when the owner of the datastore containing the link couldn't be reached.

This lead to the realization that storing linked data across personal datapods all over the internet leads to a brittle system, if not mitigated.

Linkrot is already a large problem for the current web. In 2003 an article in the washington post quoted Brewster Kahle, of internet archive fame, who estimated that the average lifespan of a web page was just 100 days. The first link I found was in an article that had made a cached copy of this article, to prevent linkrot I assume. In an ironic case of Murphy's Law, this cache was taken offline.

Read more about the problem

Based on the research mentioned below (see the "Research" section) we've formulated what is out of scope (things we won't do) and several possible solutions:

1. Forwarding in the Data
2. HTTP Status Codes
3. Meta data pod
4. Decentralized Archive/Cache

These solutions lead to the insight that two things must be created in order to solve the problem: a specification and an application.

Of the proposed solutions defined above, some require additions to one or more Solid specification. But the most basic specification is to create a new ontology that defines the semantics of forwarding.

Read more about the proposed specification changes

We're building the Solid Migrator as a set of solid application tutorials first. We found that all code examples for Solid are either outdated or use a complex toolchain setup. You can testdrive the latest version at https://pdsinterop.org/solid-migrator-app/www/

Part of the research into the problem and possible solutions revolved around linkrot and how it can be mitigated. Further details are available on separate pages:

• HTTP 300 Status and Location Header
• The Internet Archive and Wayback Machine
• DOI.org and PURL.org
• Perma.cc
• IPFS and IPNS (and NDN)
• Memento

These PDS Interop team members will be implementing the Solid Migrator roadmap:

• Auke van Slooten @poef - Research & Specification
• Ben Peachey @Potherca - Prototype Migrator
• Yvo Brevoort @ylebre - Community Outreach & Server Extensions