why is the output of `du` often so different from `du -b`

Apparent size is the number of bytes your applications think are in the file. It's the amount of data that would be transferred over the network (not counting protocol headers) if you decided to send the file over FTP or HTTP. It's also the result of cat theFile | wc -c, and the amount of address space that the file would take up if you loaded the whole thing using mmap.

Disk usage is the amount of space that can't be used for something else because your file is occupying that space.

In most cases, the apparent size is smaller than the disk usage because the disk usage counts the full size of the last (partial) block of the file, and apparent size only counts the data that's in that last block. However, apparent size is larger when you have a sparse file (sparse files are created when you seek somewhere past the end of the file, and then write something there -- the OS doesn't bother to create lots of blocks filled with zeros -- it only creates a block for the part of the file you decided to write to).

Minimal block granularity example

Let's play a bit to see what is going on.

mount tells me I'm on an ext4 partition mounted at /.

I find its block size with:

stat -fc %s .

which gives:

4096

Now let's create some files with sizes 1 4095 4096 4097:

#!/usr/bin/env bashfor size in 1 4095 4096 4097; do  dd if=/dev/zero of=f bs=1 count="${size}" status=none  echo "size     ${size}"  echo "real     $(du --block-size=1 f)"  echo "apparent $(du --block-size=1 --apparent-size f)"  echodone

and the results are:

size     1real     4096   fapparent 1      fsize     4095real     4096   fapparent 4095   fsize     4096real     4096   fapparent 4096   fsize     4097real     8192   fapparent 4097   f

So we see that anything below or equal to 4096 takes up 4096 bytes in fact.

Then, as soon as we cross 4097, it goes up to 8192 which is 2 * 4096.

It is clear then that the disk always stores data at a block boundary of 4096 bytes.

What happens to sparse files?

I haven't investigated what is the exact representation is, but it is clear that --apparent does take it into consideration.

This can lead to apparent sizes being larger than actual disk usage.

For example:

dd seek=1G if=/dev/zero of=f bs=1 count=1 status=nonedu --block-size=1 fdu --block-size=1 --apparent f

gives:

8192    f1073741825      f

Related: How to test if sparse file is supported

What to do if I want to store a bunch of small files?

Some possibilities are:

• use a database instead of filesystem: Database vs File system storage
• use a filesystem that supports block suballocation

Bibliography:

• https://serverfault.com/questions/565966/which-block-sizes-for-millions-of-small-files
• https://askubuntu.com/questions/641900/how-file-system-block-size-works

Tested in Ubuntu 16.04.

Because by default du gives disk usage, which is the same or larger than the file size. As said under --apparent-size

print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may belarger due to holes in (`sparse') files, internal fragmentation, indirect blocks, and the like