Work in progress
A quick alternative to cksfv for generating and verifying CRC32C checksum files(.sfv).
qCheck accelerates the checksum and checksum-verification process by interfacing
with the .sfv file format and using a more modern approach by utilizing both
multi-threaded parallelism and memory-mapped IO to reduce syscall overhead.
qCheck has proven itself to be about x4 to x15 times faster than
alternatives such as rhash and cksfv and reduces CRC32's algorithmic
overhead to the point of being almost entirely bounded by device IO speeds.
It is recommended to run this program on a fast NVME Solid State Drive to fully
saturate read speed.
$ git clone https://github.com/Wunkolo/qCheck.git qCheck/src
$ cmake -B qCheck/build -S qCheck/src -DCMAKE_BUILD_TYPE=Release
$ cmake --build qCheck/build --config Release
# install -Dm+x "qCheck/build/qCheck" "/usr/local/bin/qCheck"
Specs:
% inxi -IC
CPU: Topology: 10-Core model: Intel Core i9-7900X bits: 64 type: MT MCP L2 cache: 13.8 MiB
Speed: 1201 MHz min/max: 1200/4500 MHz Core speeds (MHz): 1: 1201 2: 1200 3: 1200 4: 1201 5: 1201 6: 1200 7: 1197
8: 1201 9: 1201 10: 1201 11: 1201 12: 1201 13: 1201 14: 1201 15: 1200 16: 1201 17: 1200 18: 1200 19: 1200 20: 1200
Info: Processes: 350 Uptime: 9d 14h 45m Memory: 62.51 GiB used: 19.88 GiB (31.8%) Init: systemd Shell: vim inxi: 3.0.37
% sudo nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
<snip>
/dev/nvme1n1 S~~~N~~~9~~~7~~ Samsung SSD 970 EVO 500GB 1 341.05 GB / 500.11 GB 512 B + 0 B 2B2QEXE7
Verifying an .sfv file(10GB folder full of +200MB files)
% hyperfine --runs 10 -i '../qCheck -c TestCheck.sfv' 'rhash -c TestCheck.sfv' 'cksfv -g TestCheck.sfv'
Benchmark #1: ../qCheck -c TestCheck.sfv
Time (mean ± σ): 1.494 s ± 0.198 s [User: 5.140 s, System: 0.817 s]
Range (min … max): 1.282 s … 1.740 s 10 runs
Benchmark #2: rhash -c TestCheck.sfv
Time (mean ± σ): 7.374 s ± 0.015 s [User: 5.769 s, System: 1.598 s]
Range (min … max): 7.352 s … 7.400 s 10 runs
Benchmark #3: cksfv -g TestCheck.sfv
Time (mean ± σ): 21.376 s ± 0.663 s [User: 19.848 s, System: 1.512 s]
Range (min … max): 21.084 s … 23.260 s 10 runs
Summary
'../qCheck -c TestCheck.sfv' ran
4.94 ± 0.65 times faster than 'rhash -c TestCheck.sfv'
14.31 ± 1.94 times faster than 'cksfv -g TestCheck.sfv'
Generating a .sfv file(10GB folder full of +200MB files)
% hyperfine --runs 10 "rhash SampleFiles/*" "./qCheck SampleFiles/*" "cksfv SampleFiles/*"
Benchmark #1: rhash SampleFiles/*
Time (mean ± σ): 6.234 s ± 0.145 s [User: 4.798 s, System: 1.424 s]
Range (min … max): 6.133 s … 6.632 s 10 runs
Benchmark #2: ./qCheck SampleFiles/*
Time (mean ± σ): 1.435 s ± 0.105 s [User: 5.146 s, System: 0.610 s]
Range (min … max): 1.289 s … 1.590 s 10 runs
Benchmark #3: cksfv SampleFiles/*
Time (mean ± σ): 20.896 s ± 0.074 s [User: 19.611 s, System: 1.271 s]
Range (min … max): 20.778 s … 20.996 s 10 runs
Summary
'./qCheck SampleFiles/*' ran
4.34 ± 0.33 times faster than 'rhash SampleFiles/*'
14.56 ± 1.07 times faster than 'cksfv SampleFiles/*'