🏠Home  •  🎇Features  •  🔨Installation  •  🚜Usage  •  💻CLI  •  💡Examples

🤖Jinja2 API  •  ✅Requirements  •  🐳Docker  •  🚸Gotchas

	Status	Stable	Unstable
Master
Develop

Check the progress of a long running rsync operation, by hashing chunks of the files.

• Can use any {md5sum/xxhash}-compatible CLI to hash the files.
• 🐳🌊🖥️ Docker Image (See README: Docker Image).

# Install from pypi (https://pypi.org/project/rsynccheck/)
pip install rsynccheck

# Install from git (https://github.com/realazthat/rsynccheck)
pip install git+https://github.com/realazthat/[email protected]

Example:

# Generate the audit.yaml file.
python -m rsynccheck.cli \
  hash \
  --ignorefile ".gitignore" \
  --ignoreline .trunk --ignoreline .git \
  --audit-file ".deleteme/check-changes-audit.yaml" \
  --progress none \
  --chunk-size "${CHUNK_SIZE}" \
  --directory "${SRC_DIRECTORY}"

# Check the audit.yaml file on the other machine.
python -m rsynccheck.cli \
  audit \
  --audit-file ".deleteme/check-changes-audit.yaml" \
  --progress none \
  --output-format table \
  --mismatch-exit 0 \
  --directory "${DST_DIRECTORY}"

Screenshot in terminal:

• Linux-like environment
  • Why: Uses pexpect.spawn().
• Python 3.8+
  • Why: Some dev dependencies require Python 3.8+.

• WSL2 Ubuntu 20.04, Python 3.8.0.
• Ubuntu 20.04, Python 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, tested in GitHub Actions workflow (build-and-test.yml).

Docker images are published to ghcr.io/realazthat/rsynccheck at each tag.

NOTE: You can't use a custom hashing command with the Docker image, because it isn't available inside the container. xxhash is installed in the image.

# Use the published images at ghcr.io/realazthat/rsynccheck.
# Generate the audit.yaml file.
# /data in the docker image is the working directory, so paths are simpler.
docker run --rm --tty \
  -v "${PWD}:/data" \
  ghcr.io/realazthat/rsynccheck:v0.1.0 \
  hash \
  --ignorefile ".gitignore" \
  --ignoreline .trunk --ignoreline .git \
  --audit-file ".deleteme/check-changes-audit.yaml" \
  --progress none \
  --chunk-size "${CHUNK_SIZE}" \
  --directory "${SRC_DIRECTORY}"

# Check the audit.yaml file on the other machine.
docker run --rm --tty \
  -v "${PWD}:/data" \
  ghcr.io/realazthat/rsynccheck:v0.1.0 \
  audit \
  --audit-file ".deleteme/check-changes-audit.yaml" \
  --progress none \
  --output-format table \
  --mismatch-exit 0 \
  --directory "${DST_DIRECTORY}"

If you want to build the image yourself, you can use the Dockerfile in the repository.

docker build -t my-rsynccheck-image .

# Generate the audit.yaml file.
# /data in the docker image is the working directory, so paths are simpler.
docker run --rm --tty \
  -v "${PWD}:/data" \
  my-rsynccheck-image \
  hash \
  --ignorefile ".gitignore" \
  --ignoreline .trunk --ignoreline .git \
  --audit-file ".deleteme/check-changes-audit.yaml" \
  --progress none \
  --chunk-size "${CHUNK_SIZE}" \
  --directory "${SRC_DIRECTORY}"

# Check the audit.yaml file on the other machine.
docker run --rm --tty \
  -v "${PWD}:/data" \
  my-rsynccheck-image \
  audit \
  --audit-file ".deleteme/check-changes-audit.yaml" \
  --progress none \
  --output-format table \
  --mismatch-exit 0 \
  --directory "${DST_DIRECTORY}"

We use SemVer for versioning. For the versions available, see the tags on this repository.

This project is licensed under the MIT License - see the ./LICENSE.md file for details.

• Colorful CLI help: hamdanal/rich-argparse.
• pathspec, for gitignore support: cpburnz/python-pathspec.
• humanize: jmoiron/humanize.
• xxhash: Cyan4973/xxHash.
• pydantic: pydantic/pydantic.

Not complete, and not necessarily up to date. Make a PR (contributions) to insert/modify.

• For running pre.sh (Linux-like environment).

  • From ./.github/dependencies.yml, which is used for the GH Action to do a fresh install of everything:

    bash: scripts.
    findutils: scripts.
    grep: tests.
    xxd: tests.
    git: scripts, tests.
    xxhash: scripts (changeguard).
    rsync: out-of-directory test.
    expect: for `unbuffer`, useful to grab and compare ansi color symbols.
    jq: dependency for [yq](https://github.com/kislyuk/yq), which is used to generate
      the README; the README generator needs to use `tomlq` (which is a part of `yq`)
      to query `pyproject.toml`.
    unzip: scripts (pyenv).
    curl: scripts (pyenv).
    git-core: scripts (pyenv).
    gcc: scripts (pyenv).
    make: scripts (pyenv).
    zlib1g-dev: scripts (pyenv).
    libbz2-dev: scripts (pyenv).
    libreadline-dev: scripts (pyenv).
    libsqlite3-dev: scripts (pyenv).
    libssl-dev: scripts (pyenv).
    libffi-dev: bdist_wheel (otherwise `pip install .` fails). If installing pyenv, this
      must be installed _first_.
    

  • Requires pyenv, or an exact matching version of python as in ./.python-version (which is currently 3.8.0).

  • act (to run the GH Action locally):

    • Requires nodejs.
    • Requires Go.
    • docker.

  • Generate animation:

    • docker

  • docker (for building the docker image).

1. (Optionally) Fork the develop branch.
2. Stage your files: git add path/to/file.py.
3. bash ./scripts/pre.sh, this will format, lint, and test the code.
4. git status check if anything changed (generated ./README.md for example), if so, git add the changes, and go back to the previous step.
5. git commit -m "...".
6. Make a PR to develop (or push to develop if you have the rights).

These instructions are for maintainers of the project.

1. In the develop branch, run bash ./scripts/pre.sh to ensure everything is in order.
2. In the develop branch, bump the version in ./pyproject.toml, following semantic versioning principles. Also modify the last_release and last_stable_release in the [tool.rsynccheck-project-metadata] table as appropriate. Run bash ./scripts/pre.sh to ensure everything is in order.
3. In the develop branch, commit these changes with a message like "Prepare release X.Y.Z". (See the contributions section above).
4. Merge the develop branch into the master branch: git checkout master && git merge develop --no-ff.
5. master branch: Tag the release: Create a git tag for the release with git tag -a vX.Y.Z -m "Version X.Y.Z".
6. Publish to PyPI: Publish the release to PyPI with bash ./scripts/deploy-to-pypi.sh.
7. Push to GitHub: Push the commit and tags to GitHub with git push && git push --tags.
8. The --no-ff option adds a commit to the master branch for the merge, so refork the develop branch from the master branch: git checkout develop && git merge master.
9. Push the develop branch to GitHub: git push origin develop.