parallel check md5 file
Assuming checksums.md5 has the format:
d41d8cd98f00b204e9800998ecf8427e My file name
Run:
cat checksums.md5 | parallel --pipe -N1 md5sum -c
If your files are small: -N100
If that does not speed up your processing make sure your disks are fast enough: md5sum can process 500 MB/s. iostat -dkx 1
can tell you if your disks are a bottleneck.
You need option --pipe
. In this mode parallel splits stdin into blocks and supplies each block to the command via stdin, see man parallel
for details:
cat checksums.md5 | parallel --pipe md5sum -c -
By default size of the block is 1 MB, can be changed with --block
option.