Accessing wireless interface (802.11) at MAC layer (Linux)

802.11 is layer 2 (and 1) protocol specification. It was designed in a way, which allows higher-layer protocols to treat it as Ethernet network. Addressing and behaviour is generally the same. So for a layer 3 protocol you should not be concerned about 802.11 at all and write your code as if you were expecting it to run on Ethernet network.

To make it work you should first connect to a wireless network of some sort (which is conceptually equal to plugging a wire into a Ethernet card). Here you may choose ad-hoc (aka IBSS) or infrastructural (aka BSS) network (or PBSS once 802.11ad is approved ;).

Operating cards without any sort of association with network (just spitting out packets on air) is not a good idea for a couple of reasons. Most importantly it's very hardware dependent and unreliable. You can still do it using RF mon (AKA monitor mode) interface on one side and packet injection (using radiotap header) on the other but I don't recommend that. Even if you have a set of identical cards you'll most likely encounter hard to explain and random behaviour at some point. 802.11 NICs are just not designed for this kind of operation and keep different mount of state inside firmware (read about FullMAC vs. SoftMAC cards). Even SoftMAC cards differ significantly. For example theoretically in monitor mode, as you said, card should not ACK received packets. There are cards though that will ACK received frame anyway, because they base their decision exclusively on the fact that said frame is addressed to them. Some cards may even try to ACK all frames they see. Similar thing will happen with retransmissions: some cards will send injected packet only once (that's how it should work). In other NICs, retransmissions are handled by hardware (and firmware) and driver cannot turn it off, so you will get automatic retransmission even with injected data.

Sticking with layer 3 and using existing modes (like ad hoc), will give you all capabilities you want and more (QoS etc.). Ethernet frame that you send to interface will be "translated" by the kernel to 802.11 format with QoS mapping etc.

If you want to find out about MAC behaviour in various modes you'll have to either read the mac80211 code or 802.11 standard itself. http://linuxwireless.org wiki my help you with a few things, but kernel hackers are usually to busy to write documentation other than comments in the code ;)

L3 protocol implementation itself can be also done either in kernel or user mode (using raw sockets). As usual kernel-side will be harder to do, but more powerful.

Because you want to create own network layer protocol (replacement for IP), the keyword is: "raw ethernet socket". So ignore "Raw IP socket" stuff.

This is where to start:

int sockfd = socket( PF_PACKET, SOCK_RAW, htons(XXX) ); 

Correct man page is: packet(7).

Find more information by googling with the keyword.One quite complete example here.

Edit: The link to the example seems to be currently broken: another examples

Probably you want something like libpcap.

Libpcap allows you to read/inject raw packets from/into a network interface.