64 bit large mallocs
malloc tries to allocate a contiguous memory range, and this will initially be in real memory simply due to how swap memory works (at least as far as I remember). It could easily be that your OS sometimes can't find a contiguous block of 10gb of memory and still leave all the processes that require real memory in RAM at the same time (at which point your malloc will fail).
Do you actually require 10gb of contiguous memory, or would you be able to wrap a storage class/struct around several smaller blocks and use your memory in chunks instead? This relaxes the huge contiguous requirement and should also allow your program to use the swap file for less used chunks.
Have you tried using VirtualAlloc() and VirtualFree() directly? This may help isolate the problem.
- You'll be bypassing the C runtime heap and the NT heap.
- You can reserve virtual address space and then commit it. This will tell you which operation fails.
If the virtual address space reservation fails (even though it shouldn't, judging from what you've said), Sysinternals VMMap may help explain why. Turn on "Show free regions" to look at how the free virtual address space is fragmented.