Memory usage of neural network, Keras
You are correct, this is due to the number of filters in conv1. What you must compute is the memory required to store the activations:
As shown by your model.summary(), the output size of this layer is (None, 1751, 480, 1024). For a single image, this is a total of 1751*480*1024 pixels. As your image is likely in float32, each pixel takes 4 bytes to store. So the output of this layer requires 1751*480*1024*4 bytes, which is around 3.2 GB per image just for this layer.
If you were to change the number of filters to, say, 64, you would only need around 200 MB per image.
Either change the number of filters or change the batch size to 1.