How to make my application be considered as a communication program in Windows
I believe the communication apps are implementing something related to the IAudioVolumeDuckNotification interface. The C++ example provided also references WM_VOLUME_DUCK and WM_VOLUME_UNDUCK windows messages which may be enough (but don't appear to be google-able or documented).
UPDATE
The .NET Core Audio API project on CodePlex appears to provide a .NET wrapper.
UPDATE
A sample C++ implementation of IAudioVolumeDuckNotification as well as an example of notifying the ducked state can be found in the MSDN DuckingMediaPlayer sample application. It actually works (I tried it). A combination of PInvoke'ing a couple methods and the .NET wrapper for the interface should be enough to get you on your way.