Using dlopen() on an executable

You can't open executables as libraries. The entry point of an executable will attempt to re-initialize the C library, and take over the brk pointer. This will corrupt your malloc heap. Additionally, the executable is likely to be mapped at a fixed address with no relocations, and if this address overlaps with anything already loaded, it's not possible to map it for that reason as well.

You need to refactor the other program into a library, or add a RPC interface to the other program.

Note that this does not necessarily apply for PIE executables. However, unless the executable is specifically designed for being dlopen()ed, this is unsafe, as main() will not be run, and any initialization done in main() therefore will not occur.

On some ELF systems (notably Linux), you can dlopen() PIE executables. When using GCC, just compile the executable with -fpie or -fPIE, and link it with -pie, and export the appropriate symbols using --dynamic-list or -rdynamic (explained in more detail in this other SO answer.

Tool here to do precisely that, handles ASLR/PIE and non-ASLR/PIE. Compiles on x86, ARM and MIPS (32 bit only). Edit the Makefile to set ARCH param.

http://rtfc.org.uk/cliapi.html

It's my tool but it seems to do what you want. Let me know if it doesn't work for you.

I appreciate how late I am to this party, but hey.