Having windows Azure A8 nodes with InfiniBand support how to send N bytes from one and receive on another?

I agree with Hristo's comment that it'll be MUCH easier to use a higher level API's that MPI provide, rather than a "native" IB library.
And just to clarify, MPI does not impose Master-Slave. Once all the processes are up and have a communicator, you have all the flexibility in the world. Anybody can send data to anybody. And with MPI 2.0 you have one-sided communication, where one worker can essentially reach into another's memory.

...I cant find any API for C/C++ or .Net that would allow to use InfiniBand as transport for my application. So my question is simple how to use InfiniBand to connect to other nodes and send data packets to them and receive on other end?

The C API for direct access to InfiniBand is known as 'verbs'.

Among the numerous resources online to introduce this topic, I found http://blog.zhaw.ch/icclab/infiniband-an-introduction-simple-ib-verbs-program-with-rdma-write/ to be relatively approachable.

The ultimate authority on InfiniBand software is OpenFabrics. The OFED website links docs and downloads.

I noticed under "OFS for Windows" that there is a link to Overview of Network Direct Kernel Provider Interface (NDKPI), which might meet your needs, but I have never used it because I do not use Windows.