io_submit waits for all oracle dbwriter I/Os
The problem is that when Linux scans the LUNs, the LUN advertises itself with “Write cache enabled”. This tells Linux it has to use Force Unit Access to avoid data loss in case of power loss in the cache since Oracle opens the LUN with O_SYNC (or O_DSYNC?). This is based on many presumptions – the cache is in RAM, is volatile, etc. – but let’s just accept that. FUA is bad news when it comes to performance. It also defeats the parallel issuing of async I/Os.
It turns out the array has a setting that tells it whether to advertise a Write Back Cache to the Linux server or not. It doesn’t change how the array operates; it just changes how it appears to the host. By changing the WBC setting on the array to Disabled, the Linux host prints the line “Write cache disabled” when it scans the LUN, and now async writes behave as normal.