Before you start
Associated forum post URL
No response
rclone version
rclone v1.74.3
- os/version: ubuntu 25.10 (64 bit)
- os/kernel: 6.17.0-40-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.26.4
- go/linking: static
- go/tags: none
What problem are you trying to solve?
When syncing from a bucket that is being written to at the same time, rclone often fails the whole job because a source object is deleted between the listing phase and the transfer phase.
rclone lists the source and queues an object for transfer. Before the transfer runs, the object is deleted at the source. When rclone opens it to copy it, the source returns a 404:
failed to open source object: googleapi: got HTTP response code 404 with body: No such object: my-bucket/path/to/file.jpg
This 404 is counted as an error. Two things follow from that:
- rclone refuses to delete on the destination:
not deleting files as there were IO errors
not deleting directories as there were IO errors
- The sync returns a non-zero result, so the job is reported as failed.
This happens on every run against live buckets. In our case it is GCS to an S3 backend, scheduled daily, and around 5% of runs fail this way every week. The data is fine. The only problem is that an object listed earlier no longer exists at transfer time, which is normal for a bucket that keeps changing.
The existing flags do not solve this:
--ignore-errors: lets the deletion happen, but the job still returns a non-zero result, because the error stays counted.--retries: re-runs the whole sync. On a bucket that keeps changing, each retry hits new deleted objects, so it does not converge. It also re-lists the whole bucket every attempt, which is expensive on buckets with millions of objects.--low-level-retries: a 404 is not retryable, so this does nothing here.--min-age: reduces the window but does not remove it, since an old object can also be deleted during the run.
This was discussed on the forum and the conclusion at the time was that there is no flag for it and the workaround is to parse the log: https://forum.rclone.org/t/how-to-ignore-failed-to-copy-failed-to-open-source-object-errors/21734
How do you think rclone should be changed to solve that?
Add a new flag, off by default, for example --skip-missing-source.
When the flag is set and opening a source object fails because the object no longer exists, rclone would:
- log it as a skipped file, not an error
- not count it as an error, so destination deletions still run and the job returns success
- leave all other errors untreated, so real failures (auth, destination down, listing errors) still block deletions and fail the job as they do today
With the flag off, behaviour stays exactly as it is now.
Implementation outline, for discussion:
- The source 404 reaches
fs/operations/copy.go as a backend specific error (a *googleapi.Error for GCS, an S3 error for S3), not as fs.ErrorObjectNotFound. So matching on the open error directly is not portable.
- A portable check is to re-stat the source after the open fails: call
NewObject on the source remote. Backends translate a missing object to fs.ErrorObjectNotFound, so this works the same across GCS, S3 and others.
- If the re-stat says the object is gone and the flag is set, skip the file without counting an error. Otherwise behave as today.
This stays inside fs/ (config, operations/copy, sync) with no backend changes and no new dependency.
Getting involved
Before you start
Associated forum post URL
No response
rclone version
rclone v1.74.3
What problem are you trying to solve?
When syncing from a bucket that is being written to at the same time, rclone often fails the whole job because a source object is deleted between the listing phase and the transfer phase.
rclone lists the source and queues an object for transfer. Before the transfer runs, the object is deleted at the source. When rclone opens it to copy it, the source returns a 404:
failed to open source object: googleapi: got HTTP response code 404 with body: No such object: my-bucket/path/to/file.jpg
This 404 is counted as an error. Two things follow from that:
not deleting files as there were IO errors
not deleting directories as there were IO errors
This happens on every run against live buckets. In our case it is GCS to an S3 backend, scheduled daily, and around 5% of runs fail this way every week. The data is fine. The only problem is that an object listed earlier no longer exists at transfer time, which is normal for a bucket that keeps changing.
The existing flags do not solve this:
--ignore-errors: lets the deletion happen, but the job still returns a non-zero result, because the error stays counted.--retries: re-runs the whole sync. On a bucket that keeps changing, each retry hits new deleted objects, so it does not converge. It also re-lists the whole bucket every attempt, which is expensive on buckets with millions of objects.--low-level-retries: a 404 is not retryable, so this does nothing here.--min-age: reduces the window but does not remove it, since an old object can also be deleted during the run.This was discussed on the forum and the conclusion at the time was that there is no flag for it and the workaround is to parse the log: https://forum.rclone.org/t/how-to-ignore-failed-to-copy-failed-to-open-source-object-errors/21734
How do you think rclone should be changed to solve that?
Add a new flag, off by default, for example
--skip-missing-source.When the flag is set and opening a source object fails because the object no longer exists, rclone would:
With the flag off, behaviour stays exactly as it is now.
Implementation outline, for discussion:
fs/operations/copy.goas a backend specific error (a*googleapi.Errorfor GCS, an S3 error for S3), not asfs.ErrorObjectNotFound. So matching on the open error directly is not portable.NewObjecton the source remote. Backends translate a missing object tofs.ErrorObjectNotFound,so this works the same across GCS, S3 and others.This stays inside fs/ (config, operations/copy, sync) with no backend changes and no new dependency.
Getting involved