Large PCIe DMA Linux x86-64
Instead of limiting the amount of system memory via mem, try using CMA: https://lwn.net/Articles/486301/
Using the CMA kernel command line argument allows you to reserve a certain amount of memory for DMA operations that is guaranteed to be contiguous. The kernel will allow non-DMA processes to access that memory, but as soon as a DMA operation needs that memory, non-DMA processes will be evicted. So, I would advise not changing your mem parameter, but adding cma=4G to your cmdline. dma_alloc_coherent should automatically pull from that reserved space, but you can enable CMA debugging in your kernel config to make sure.