When is a shared library considered to be "the same" for the purpose of sharing?

The dynamic loader just does regular old open(2) and mmap(2) calls, and a memory map bumps inode refcounts the same way an open fd does. So if you do the usual atomic-file-replacement trick for the library, write out your changes in a new copy of the file and then rename(2) it over the old name, anything that's started after that point will pick up the new inode and the new contents, but running programs will keep using the old inode and the old contents.

If you modify the library in place, naturally any programs started after the write call will pick up your changes. The more interesting question is what happens to processes that already had it mapped. The answer is probably either "the system won't let you do that" or "unspecified, depends on details of the page cache implementation". Poking casually at the Linux implementation (which is what I have to hand): The glibc dynamic loader uses MAP_DENYWRITE for all its shared library maps, which is not documented but sounds like it means "make this file unmodifiable while the mapping exists". However, I can't find anything in the kernel sources that makes MAP_DENYWRITE do anything; it might be a historical vestige or some such.

It also uses MAP_PRIVATE. http://pubs.opengroup.org/onlinepubs/7908799/xsh/mmap.html says that "It is unspecified whether modifications to the underlying object done after the MAP_PRIVATE mapping is established are visible through the MAP_PRIVATE mapping." So you might or might not be able to modify a shared library image underneath a running process, depending on the details of the page cache implementation.

The dynamic loader uses mmap(2) to load shared libraries, so all the magic is really in mmap(2).

In the specific case of Linux, file mappings call a fs-specific mmap() operation, which usually is wired to mm/filemap.c:generic_file_mmap(), which sets the vm_ops.fault for the file mapping to mm/filemap.c:filemap_fault(), so the magic is delayed to page fault time. filemap_fault() tries to find the page in the page cache using find_lock_page().

mmap() → fs_file_ops.mmap() → generic_file_mmap() → file_vm_ops.fault = filemap_fault()page fault → filemap_fault() → find_lock_page()