Address sanitizing Boost.Python modules

So, I finally managed to get this to work:

$ libasan=/opt/local/libexec/llvm-4.0/lib/clang/4.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib$ python=/opt/local/Library/Frameworks/Python.framework/Versions/3.5/Resources/Python.app/Contents/MacOS/Python$ DYLD_INSERT_LIBRARIES=$libasan $python -c 'import hello_ext; print(hello_ext.greet())'===================================================================70859==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000002770 at pc 0x000108c2ef60 bp 0x7ffee6fe8c20 sp 0x7ffee6fe83c8READ of size 2 at 0x60b000002770 thread T0    #0 0x108c2ef5f in wrap_strlen (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x14f5f)    #1 0x109a8d939 in PyUnicode_FromString (Python:x86_64+0x58939)[...]

What changed? Nothing in the compilation chain, just the invocation.

Let PYDIR=/opt/local/Library/Frameworks/Python.framework/Versions/3.5: previously I was calling $PYDIR/bin/python3.5 (because /opt/local/bin/python3.5 is a symlink to it), now I call $PYDIR/Resources/Python.app/Contents/MacOS/Python.

To understand what was going on, I ran DYLD_INSERT_LIBRARIES=$libasan python3.5, and looked for its open files

$ ps  PID TTY           TIME CMD  900 ttys000    0:07.96 -zsh70897 ttys000    0:00.11 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/Resources/Python.app/Contents/MacOS/Python53528 ttys001    0:05.14 -zsh  920 ttys002    0:10.28 -zsh$ lsof -p 70897COMMAND   PID USER   FD   TYPE DEVICE   SIZE/OFF       NODE NAMEPython  70897 akim  cwd    DIR    1,4        480 8605949500 /Users/akim/src/lrde/vcsn/experiment/sanitizerPython  70897 akim  txt    REG    1,4      12988 8591019542 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/Resources/Python.app/Contents/MacOS/PythonPython  70897 akim  txt    REG    1,4    2643240 8591012758 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/PythonPython  70897 akim  txt    REG    1,4     107524 8590943656 /opt/local/lib/libintl.8.dylibPython  70897 akim  txt    REG    1,4    2097528 8590888556 /opt/local/lib/libiconv.2.dylibPython  70897 akim  txt    REG    1,4      20224 8591016920 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload/_heapq.cpython-35m-darwin.soPython  70897 akim  txt    REG    1,4     326996 8591375651 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/readline/gnureadline.cpython-35m-darwin.soPython  70897 akim  txt    REG    1,4     603008 8605907803 /opt/local/lib/libncurses.6.dylibPython  70897 akim  txt    REG    1,4     837248 8606849556 /usr/lib/dyldPython  70897 akim  txt    REG    1,4 1155837952 8606860187 /private/var/db/dyld/dyld_shared_cache_x86_64hPython  70897 akim    0u   CHR   16,0  0t2756038        667 /dev/ttys000Python  70897 akim    1u   CHR   16,0  0t2756038        667 /dev/ttys000Python  70897 akim    2u   CHR   16,0  0t2756038        667 /dev/ttys000

Obviously libasan is not here, and that's the whole problem. However, I also noticed that ps was referring to another Python than the one I ran (and, of course, it's part of the open files).

It turns out that there are several Python executables in this directory: $PYDIR/bin/python3.5 and $PYDIR/Resources/Python.app/Contents/MacOS/Python, and the first one, in one way or another, bounces to the second. If I run the second with DYLD_INSERT_LIBRARIES=$libasan

$ lsof -p 71114COMMAND   PID USER   FD   TYPE DEVICE  SIZE/OFF       NODE NAMEPython  71114 akim  cwd    DIR    1,4       480 8605949500 /Users/akim/src/lrde/vcsn/experiment/sanitizerPython  71114 akim  txt    REG    1,4     12988 8591019542 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/Resources/Python.app/Contents/MacOS/PythonPython  71114 akim  txt    REG    1,4   3013168 8604479549 /opt/local/libexec/llvm-4.0/lib/clang/4.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylibPython  71114 akim  txt    REG    1,4   2643240 8591012758 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/PythonPython  71114 akim  txt    REG    1,4    107524 8590943656 /opt/local/lib/libintl.8.dylibPython  71114 akim  txt    REG    1,4   2097528 8590888556 /opt/local/lib/libiconv.2.dylibPython  71114 akim  txt    REG    1,4     20224 8591016920 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload/_heapq.cpython-35m-darwin.soPython  71114 akim  txt    REG    1,4    326996 8591375651 /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/readline/gnureadline.cpython-35m-darwin.soPython  71114 akim  txt    REG    1,4    603008 8605907803 /opt/local/lib/libncurses.6.dylibPython  71114 akim  txt    REG    1,4    837248 8606849556 /usr/lib/dyldPython  71114 akim    0u   CHR   16,0 0t2781894        667 /dev/ttys000Python  71114 akim    1u   CHR   16,0 0t2781894        667 /dev/ttys000Python  71114 akim    2u   CHR   16,0 0t2781894        667 /dev/ttys000

\o/ libasan is there! So apparently, the first Python calls the second one, and DYLD_INSERT_LIBRARIES is not forwarded.

I currently have no idea why there are two Pythons. It does not seem to be specific to MacPorts, as it's also the case for Apple's Python.

$ cd /System/Library/Frameworks/Python.framework/Versions/2.7$ ls -l bin/python*lrwxr-xr-x  1 root  wheel      7 10 déc 08:17 bin/python -> python2lrwxr-xr-x  1 root  wheel     14 10 déc 08:17 bin/python-config -> python2-configlrwxr-xr-x  1 root  wheel      9 10 déc 08:17 bin/python2 -> python2.7lrwxr-xr-x  1 root  wheel     16 10 déc 08:17 bin/python2-config -> python2.7-config-rwxr-xr-x  1 root  wheel  43104  1 déc 21:42 bin/python2.7-rwxr-xr-x  1 root  wheel   1818 16 jul 02:20 bin/python2.7-configlrwxr-xr-x  1 root  wheel      8 10 déc 08:17 bin/pythonw -> pythonw2lrwxr-xr-x  1 root  wheel     10 10 déc 08:17 bin/pythonw2 -> pythonw2.7-rwxr-xr-x  1 root  wheel  43104  1 déc 21:42 bin/pythonw2.7$ ls -l Resources/Python.app/Contents/MacOS/Python-rwxr-xr-x  1 root  wheel  51744  1 déc 21:48 Resources/Python.app/Contents/MacOS/Python

This may be a less than ideal answer, but it should provide you with what you need.

I'm proposing you make an Xcode project to build everything (yes, I know you want to use make, that comes later)

Assuming you already know how to get a Xcode project building everything I'll skip ahead to enabling Address Sanitizer:

Product -> Scheme -> Edit Scheme will open this window:

Check the boxes for Address Sanitizer. You should only have to do this for your main Application. In my experience all the dependancies will be built appropriately.

Next Build the main application target. We will need to examine the compiler and linker invocations so we can copy any necessary flags/steps to the make files.

I've circled the 2 relevant buttons to get the output. The rightmost button will expand the compiler invocation used to build.

This shot shows both a save button (it may be easier to examine in another program) and some of the relevant flags used for building. You will have to examine the output for all targets (dependancies) to get a clear picture.

Hope this helps. FWIW the project in my example has a custom Python module written in C++ (directly using libPython, not Boost) so I know you can use Asan on those even using the System supplied Python frameworks.