Lost memory on Linux - not cached, not buffers

This memory is very likely used by the ZFS ARC cache and other ZFS related data stored in the kernel memory. The ARC cache is somewhat similar to the buffer cache so there is generally nothing to worry about it as this memory is released by ZFS should there is demand to it.

However, there is a subtle difference between buffer cache memory and ARC cache one. The first one is immediately available to allocation while the ARC cache one is not. ZFS monitors the free RAM available and when too low, it releases RAM to other consumers.

This works fine with most applications but a minority of them are either confused when a low amount of available RAM is reported, or allocate too much/too fast memory for the release process to keep up the pace properly.

That's the reason why ZFS allows to reduce the maximum size the ARC size is allowed to use.This setting is done in the /etc/modprobe.d/zfs.conf file.

For example, should you want the ARC never to exceed 32 GB, add this line:

options zfs zfs_arc_max=34359738368

To get the current ARC size and various other ARC statistics, run this command:

cat /proc/spl/kstat/zfs/arcstats

The size metric will show the current size of the ARC. Beware that other ZFS related memory areas might also take a share of RAM and won't be necessarily quickly released even when no more used. Finally, ZFS on linux is certainly less mature than the Solaris native implementation so you might be hit by a bug like this one.

Note too that due to the share storage pool design, unmounting a ZFS file system won't free any resource. You would need to export a pool for memory to be eventually released.