Dealing with large binary file (3 GB) in docker and Jenkins
The problem was mainly caused due to the following reasons:
- The default value of client_max_body_size in Ngnix server configuration was very low. Due to which, we could not upload a file of 3.6 GB so we increased this value to 4 GB.
- We run a Jetty server on our repository management system to serve out HTTP traffic so we needed to increase the time-out there for Jenkins to pull the relevant docker files from there.
This answer is mainly in the context of this specific problem. However, the question regarding how to handle such files in a better way still remains open. Moreover, it is not clear if increasing client_max_body_size to 4 GB is a good idea in general.
Relevant docs for client_max_body_size:http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size