how to send really really large json object as response - node.js with express how to send really really large json object as response - node.js with express json json

how to send really really large json object as response - node.js with express


After a lot of try, I finally decided to go with socket.io to send each config file at a time rather than all config files at once. This solved the problem of out of memory which was crashing my server. thanks for all your help


Try to use streams. What you need is a readable stream that produces data on demand. I'll write simplified code here:

var Readable = require('stream').Readable;var rs = Readable();rs._read = function () {    // assuming 10000 lines of config fits in memory    rs.push({config:<10,000 lines of config>);};rs.pipe(res);


You can try increasing the memory node has available with the --max_old_space_size flag on the command line.

There may be a more elegant solution. My first reaction was to suggest using res.json() with a Buffer object rather than trying to send the entire object all in one shot, but then I realize that whatever is converting to JSON will probably want to use the entire object all at once anyway. So you will run out of memory even though you are switching to a stream. Or at least that's what I would expect.