STL and release/debug library mess

The debug mode referenced here has nothing to do with debug or release build of your application. The STL debug mode is activated with -D_GLIBCXX_DEBUG and is a special checking mode.

It is very unlikely that the 3rd party library was in fact compiled with STL checking mode, but if it was, it would likely very promptly mention that your code should also be compiled with -D_GLIBCXX_DEBUG.

If the 3rd party library was not built with checking STL, then it is compatible with your code regardless of whether you are doing optimized or debug build.

Since you state that debug build of your code linked with optimized build of 3rd party library causes a crash, that crash is most likely caused by a bug in your code (or possibly by a bug in 3rd party library).

Valgrind and GDB are your friends.

I believe that you have misread the documentation at the link you provide. In particular, you've misunderstood its purpose -- that section is entitled "Goals", and describes a number of hypothetical designs for a C++ debug library and the consequences of those designs in order to explain the actual design choices that were made. The bits of text that follow the lines you quoted are describing the chaos that would result from a hypothetical implementation that had separate designs for release-mode and debug-mode strings. It goes on to say:

For this reason we cannot easily provide safe iterators for the std::basic_string class template, as it is present throughout the C++ standard library.

(Or, rephrasing that, providing a special "debug" version of string iterators is impossible.)

...

With the design of libstdc++ debug mode, we cannot effectively hide the differences between debug and release-mode strings from the user. Failure to hide the differences may result in unpredictable behavior, and for this reason we have opted to only perform basic_string changes that do not require ABI changes. The effect on users is expected to be minimal, as there are simple alternatives (e.g., __gnu_debug::basic_string), and the usability benefit we gain from the ability to mix debug- and release-compiled translation units is enormous.

In other words, the design of the debug and release modes in GCC's libstdc++ has rejected this hypothetical implementation with separate designs for the strings, specifically in order to allow cross-mode linking of the sort that you are worrying about how to avoid.

Thus, you should not have problems with compiling your library once, without -D_GLIBCXX_DEBUG (or with it, if for some reason you prefer), and linking it with either mode of your application. If you do have problems, it is due to a bug somewhere. [But see edit below! This is specific to std::string, not other containers!]

Edit: After this answer was accepted, I followed up in answering the follow-up question at std::vector<std::string> crash, and realized that the conclusion of this answer is incorrect. GCC's libstdc++ does clever things with strings to support "Per-use recompilation" (in which all uses of a given container object must be compiled with the same flags, but uses of the same container class within a program need not be compiled with the same flags), but that is not the same thing as complete "Per-unit compilation" that would provide the cross-linking ability you need. In particular, the documentation says of that cross-linking ability,

We believe that this level of recompilation is in fact not possible if we intend to supply safe iterators, leave the program semantics unchanged, and not regress in performance under release mode....

Thus, if you're passing containers across your library interface, you will need two separate libraries. Honestly, for this situation I've found that the easiest solution is just to install the two libraries into different directories (one for each variant -- and you'll want both to be separate from your main library directory). Alternately, you can rename the debug library file and then install it manually.

As a further suggestion -- you're presumably not running this in debug mode very often. It may be worth only compiling and linking the debug version statically into your application, so you don't have to worry about installing multiple dynamic libraries and keeping them straight at runtime.

Give the debug and release versions of the DLL different names and link the correct one through library dependency. Your application then wont start unless it finds the correct DLL.