Not understanding dd command arguments at all

I'm not yet sure why, but using this method will not fill up an entire block before saving it. Try:

perl -e 'print "testing\n" while 1' | dd of=output.txt bs=10K count=10K iflag=fullblock10240+0 records in10240+0 records out104857600 bytes (105 MB) copied, 2.79572 s, 37.5 MB/s

The iflag=fullblock seems to force dd to accumulate input until the block is full, although I'm not sure why this is not the default, or what it actually does by default.

To see what's going on, let's look at the output of strace for a similar invocation:

execve("/bin/dd", ["dd", "of=X", "bs=1M", "count=2"], [/* 72 vars */]) = 0…read(0, "testing\ntesting\ntesting\ntesting\n"..., 1048576) = 69632write(1, "testing\ntesting\ntesting\ntesting\n"..., 69632) = 69632read(0, "testing\ntesting\ntesting\ntesting\n"..., 1048576) = 8192write(1, "testing\ntesting\ntesting\ntesting\n"..., 8192) = 8192close(0)                                = 0close(1)                                = 0write(2, "0+2 records in\n0+2 records out\n", 31) = 31write(2, "77824 bytes (78 kB) copied", 26) = 26write(2, ", 0.000505796 s, 154 MB/s\n", 26) = 26…

What happens is that dd makes a single read() call to read each block. This is appropriate when reading from a tape, which is what dd was originally mainly used for. On tapes, read really reads a block. When reading from a file, you have to be careful not to specify a too large block size, or else the read will be truncated. When reading from a pipe, it's worse: the size of the block that you read will depend on the speed of the command producing the data.

The moral of the story is not to use dd to copy data, except with safe, small blocks. And never from a pipe except with bs=1.

(GNU dd has a fullblock flag to tell it to behave decently. But other implementations don't.)

My best guess is that dd reads from the pipe and when it's empty it assumes that it read the whole block. The results are quite inconsistent:

$ perl -e 'print "testing\n" while 1' | dd of=X bs=1M count=1000+100 records in0+100 records out413696 bytes (414 kB) copied, 0.0497362 s, 8.3 MB/suser@andromeda ~$ perl -e 'print "testing\n" while 1' | dd of=X bs=1M count=1000+100 records in0+100 records out409600 bytes (410 kB) copied, 0.0484852 s, 8.4 MB/s