Multiple paginated GET API calls in parallel/async in Node

I was able to get it working be creating a function with callback.

export function getFilesList() {  const foldersURL: any[] = [];  getFoldersFromRepo().then((response) => {    const values = response.values;    values.forEach((value: any) => {    //creating API URL for each folder in the repo      const URL = 'https://bitbucket.abc.com/stash/rest/api/latest/projects/'                   + value.project.key + '/repos/' + value.slug + '/files?limit=1000';      foldersURL.push(URL);        });    return foldersURL;      }).then((res) => {    // console.log('Calling all the URLS in parallel');    async.map(res, (link, callback) => {       const options = {         url: link,         auth: {           password: 'password',           username: 'username',         },       };      const myarray = [];// This function will consolidate response till the last Page per API.      consolidatePaginatedResponse(options, link, myarray, callback);     }, (err, results) => {       console.log('In err, results function');       if (err) {         return console.log(err);       }       //Consolidated results after all API calls.       console.log('results', results);     });  })   .catch((error) => error);}function consolidatePaginatedResponse(options, link, myarray, callback) {  request(options, (error, response, body) => {    const content = JSON.parse(body);    content.link = options.url;    myarray.push(content);    if (content.isLastPage === false) {      options.url = link + '&start=' + content.nextPageStart;      consolidatePaginatedResponse(options, link, myarray, callback);    } else {// Final response after consolidation per API      callback(error, JSON.stringify(myarray));    }  });}

I think the best way is to wrap it in a old school for loop (forEach doesn't work with async, since it's synchronous and it will cause all the requests to be spawn at the same time).

What I understood is that you do some sort of booting query where you get the values array and then you should iterate among the pages. Here some code, I didn't fully grasp the APIs so I'll give a simplified (and hopefully readable) answer, you should be able to adapt it:

export async function getFilesList() {    logger.info(`Fetching all the available values ...`);    await getFoldersFromRepo().then( async values => {        logger.info("... Folders values fetched.");        for (let i = 0; ; i++ ) {            logger.info( `Working on page ${i}`);            try {                // if you are using TypeScript, the result is not the promise but the succeeded value already                const pageResult: PageResult = await yourPagePromise(i);                if (pageResult.isLastPage) {                    break;                }            } catch(err) {                console.err(`Error on page ${i}`, err);                break;            }        }        logger.info("Done.");    });    logger.info(`All finished!`);}

The logic behind is that first getFoldersFromRepo() returns a promise which returns the values, and then I sequentially iterate on all available pages through the yourPagePromise function (which returns a promise). The async/await construct allows to write more readable code, rather then having a waterfall of then().

I'm not sure it respects your APIs specs, but it's the logic you can use as foundation! ^^