Running multiple task using web worker in Nodejs

As I'm sure you have read, the node is single-threaded, so running transactions in parallel is not going to work, even with worker threads as they are not designed to run in parallel.

A worker thread is more for longer, more process intense functions that you want to pass off and not block the main event loop, so if you think of it in terms of uploading and processing an image.. well we don't really want to hang up the entire event loop while the image is processed, so we can pass it off to a worker thread and it will tell the event loop when it's done, and it will return the response.

I think what you may be looking to do is just create a promise, so you would have a promise and say an array of the JSON file name like ["file1.JSON", "file2.JSON"] Then in your promise you would loop over, read the contents and 'return' the JSON object, insert or concat the main array variable.

Once the promise resolves, you would use the

.then(()=>{ //Do you processing of the full array })

Here's an example with a library (node-worker-threads-pool).

Thread/worker management is a complex endeavor, and I would not recommend trying to have some generic solution. Even the library I'm suggesting may not be correct.

// sample.jsconst { StaticPool } = require('node-worker-threads-pool');const start = async function () {    const staticPool = new StaticPool({        size: 4,        task: async function(n) {            const sleep = async function (ms) {                return new Promise(resolve => setTimeout(resolve, ms));            }            console.log(`thread ${n} started`);            await sleep(1000 * n);            return n + 1        }    });    // start 4 workers, each will run asynchronously and take a longer time to finish    for (let index = 0; index < 4; index++) {        staticPool.exec(index)            .then((result) => {                console.log(`result from thread pool for thread ${index}: ${result}`);            })            .catch((err) => console.error(`Error: ${err}`));    }}start();

I ran this in npm using node sample.js

As discussed in the other answer, it may not be useful (in terms of performance) to do this, but this example shows how it can be done.

The library also has examples where you give the tasks specific work.