Minimizing memory footprint in C programs

If your program has so many malloc and free calls that you get this kind of fragmentation, using mmap for each allocation will be hopelessly slow. Instead you need to measure what's causing the memory fragmentation and fix it. First I would use a tool like valgrind to make sure it's not actually a memory leak/corruption issue causing the excessive memory usage. Then, to confirm that the cause of the problem is fragmentation, I would wrap all calls to malloc and free with your own wrapper that increments and decrements a "total allocated byte count" variable so you can, at any point, compare the theoretical and actual memory consumption.

If it turns out fragmentation is the problem, a good first step is looking at why you're making so many small, short-lived allocations. If you can eliminate them and instead allocate all the memory a particular task/data object will need in a single block, then chop it up yourself, you'll not only get rid of the worst fragmentation but also improve the performance of your code quite a bit. Each call to malloc or free incurs a good bit of overhead, especially in a threaded environment where synchronization/locking is necessary. Keeping all associated data together in a single allocated block can also reduce or eliminate the need to write special code to free structures containing pointers; a single call to free may often suffice (though for the sake of keeping the implementation opaque you should still wrap this with a foo_free function).

Unless you are allocating fixed sized blocks - or performing some sort of periodic garbage collection - you memory footprint is likely to grow beyond what is required due to fragmentation.

free() cannot return memory to the OS unless full pages not used.

My suggestion would be to use a slab allocation scheme. Pre-allocate several pools of memory. Each pool will be used with objects of similar size (ideally identical size). This way you avoid fragmentation and keep your RSS footprint constant (albeit bigger than absolutely necessary due to preallocation).