How can I limit the cache used by copying so there is still memory available for other caches?

The nocache command is the general answer to this problem! It is also in Debian and Ubuntu 13.10 (Saucy Salamander).

Thanks, Peter, for alerting us to the --drop-cache" option in rsync. But that was rejected upstream (Bug 9560 – drop-cache option), in favor of a more general solution for this: the new "nocache" command based on the rsync work with fadvise.

You just prepend "nocache" to any command you want. It also has nice utilities for describing and modifying the cache status of files. For example, here are the effects with and without nocache:

$ ./cachestats ~/file.mp3pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]$ ./nocache cp ~/file.mp3 /tmp$ ./cachestats ~/file.mp3pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]\$ cp ~/file.mp3 /tmp$ ./cachestats ~/file.mp3pages in cache: 1945/1945 (100.0%)  [filesize=7776.2K, pagesize=4K]

So hopefully that will work for other backup programs (rsnapshot, duplicity, rdiff-backup, amanda, s3sync, s3ql, tar, etc.) and other commands that you don't want trashing your cache.

Kristof Provost was very close, but in my situation, I didn't want to use dd or write my own software, so the solution was to use the "--drop-cache" option in rsync.

I have used this many times since creating this question, and it seems to fix the problem completely. One exception was when I am using rsync to copy from a FreeBSD machine, which doesn't support "--drop-cache". So I wrote a wrapper to replace the /usr/local/bin/rsync command, and remove that option, and now it works copying from there too.

It still uses huge amount of memory for buffers and seems to keep almost no cache, but it works smoothly anyway.

$ free             total       used       free     shared    buffers     cachedMem:      24731544   24531576     199968          0   15349680     850624-/+ buffers/cache:    8331272   16400272Swap:      4194300     602648    3591652

The kernel can not know that you won't use the cached data from copying again. This is your information advantage.

But you could set the swapiness to 0: sudo sysctl vm.swappiness=0. This will cause Linux to drop the cache before libraries, etc. are written to the swap.

It works nice for me too, especially very performant in combination with huge amount of RAM (16-32 GB).