Piping stream from busboy to request post Piping stream from busboy to request post express express

Piping stream from busboy to request post


The problem here is that you need to provide 'Content-Length' for the multipart upload manually, because request (and underlying form-data) can't figure it out by themselves.So request sends invalid Content-Length: 199 (the same for any incoming file size), which breaks the java multipart parser.

There are multiple workarounds:

1) Use incoming request 'Content-Length'

request.post({  url: server.baseURL + 'api/data',  formData: {    file: {      value: fileStream,      options: {        knownLength: req.headers['content-length']      }    }  }}, function (err, r, body) {  // Do rendering stuff, handle callback})

This will produce a bit incorrect request though, because incoming length includes other upload fields and boundaries, but busboy was able to parse it w/o any complaints

2) Wait until file is completely buffered by the node app then send it to java

var concat = require('concat-stream')req.busboy.on('file', function (fieldName, fileStream, fileName, encoding, mimeType) {  fileStream.pipe(concat(function (fileBuffer) {    request.post({      url: server.baseURL + 'api/data',      formData: {        file: fileBuffer      }    }, function (err, r, body) {      // Do rendering stuff, handle callback    })  }))})

This will increase app memory consumption, so you needed to be careful and consider using busboy limits

3) Buffer file to disk before uploading (just for the reference)

  • express + multer - I recommend using express for webservers, it makes things more manageable, and multer is based on the busboy
  • formidable


If possible, send a custom header with exactly size file (bytes). The header always can be read before handle payload stream. Use this instead of content-length header of previous answer, because that sometimes doesn't works (with small files, i guess, but i cannot ensure that works with large files).


For the answer of Afanasii Kurakin

request.post({  url: server.baseURL + 'api/data',  formData: {    file: {      value: fileStream,      options: {        knownLength: req.headers['content-length']      }    }  }}, function (err, r, body) {  // Do rendering stuff, handle callback})

You should change from req.headers['content-length'] to the real file size, normally the content-length from header is bigger than the file size. I got pain because of the content-length and after using the file size, everything worked perfectly.