Why I cannot override search path of dynamic libraries with LD_LIBRARY_PATH?

From http://man7.org/linux/man-pages/man8/ld.so.8.html

When resolving shared object dependencies, the dynamic linker first inspects each dependency string to see if it contains a slash (this can occur if a shared object pathname containing slashes was specified at link time). If a slash is found, then the dependency string is interpreted as a (relative or absolute) pathname, and the shared object is loaded using that pathname.

If a shared object dependency does not contain a slash, then it is searched for in the following order:

o (ELF only) Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not exist. Use of DT_RPATH is deprecated.

o Using the environment variable LD_LIBRARY_PATH. Except if the executable is a set-user-ID/set-group-ID binary, in which case it is ignored.

o (ELF only) Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present.

o From the cache file /etc/ld.so.cache, which contains a compiled list of candidate shared objects previously found in the augmented library path. If, however, the binary was linked with the -z nodeflib linker option, shared objects in the default paths are skipped. Shared objects installed in hardware capability directories (see below) are preferred to other shared objects.

o In the default path /lib, and then /usr/lib. (On some 64-bit archiectures, the default paths for 64-bit shared objects are /lib64, and then /usr/lib64.) If the binary was linked with the -z nodeflib linker option, this step is skipped.

• with readelf readelf -d libfftw3_mpi.so you can check if your lib contains such a attribute in the dynamic section.

• with export LD_DEBUG=libs you can debug the search path used to find your libs

• with chrpath -r<new_path> <executable> the rpath can be changed

I see two possible reasons for this.

First, libfftw3_mpi.so may be linked with /usr/lib64/ as RPATH. In that case, providing LD_LIBRARY_PATH will have no effect. To check if it is your case, run readelf -d libfftw3_mpi.so | grep RPATH and see if it has /usr/lib64/ as a library path. If it does, use chrpath utility to change or remove it.

Alternatively, you may be running a system that does not support LD_LIBRARY_PATH at all (like HP-UX).