How to stream file using curl from one server to another(limited server resources) How to stream file using curl from one server to another(limited server resources) linux linux

How to stream file using curl from one server to another(limited server resources)


Another SO user asked a similar question about curl posts from stdin. See use pipe for curl data.

Once you are able to post your upload stream from the output of the first curl process's standard output, if you are running out of memory because you are downloading faster than you can upload, have a look at the mbuffer utility. I haven't used it myself, but it seems to be designed for exactly this sort of problem.

Lastly, if all else fails, I guess you could use curl's --limit-rate option to lock the transfer rates of the upload and download to some identical and sustainable values. This potentially underutilizes bandwidth, and won't scale well with multiple parallel download/upload streams, but for some one-off batch process, it might be good enough.