How to ensure all records are retrieved from Airtable before Promise resolves in Node.js async/await?

Other Questions Airtable
1

I’m working with Airtable API in Node.js and facing an issue with async/await operations. I need to fetch all records from my Airtable base before proceeding to the next step in my workflow.

The problem is that my Promise seems to resolve before all records are actually processed. I want to wait until every single record has been retrieved and processed before moving forward.

Here’s my main function that calls the Airtable data retrieval:

var status = 0;

async function MainProcess(){
    status = await dataRetriever.fetchAllData();
    console.log("Status value is ", status);
    if(status == 1){
        status = await nextModule.fetchAllData(); 
        console.log("Status value is ", status);
    }
}

And here’s my Airtable data fetching function:

async function fetchAllData()
{
    return new Promise((resolve, reject) => {
            console.log("Starting data retrieval process");
            var count = 0;
            let result = base("User Data").select({
            }).eachPage(function page(results, getNextPage) {
                results.forEach(function(result) {
                    let userID = result.get("UserID");
                    let userEmail = result.get("User Email");
                    ProcessUserData(userID, userEmail);
                    count++;
                });
                getNextPage();
            }, function done(error) {
                if (error) {
                    console.error(error);
                    reject("Data retrieval failed");
                } else {
                    resolve(1);
                }
            });
      });
}

How can I make sure that all records are completely fetched and processed before the Promise resolves? The done callback should only trigger after all pages have been processed, but it seems like it’s resolving too early.

2

The problem is ProcessUserData is probably async but you’re not waiting for it to finish. The done callback fires when all pages are fetched, not when processing is done.

I hit this exact issue last year with a similar Airtable setup. Here’s my fix:

async function fetchAllData() {
    return new Promise((resolve, reject) => {
        console.log("Starting data retrieval process");
        const processingPromises = [];
        
        base("User Data").select({
        }).eachPage(function page(results, getNextPage) {
            results.forEach(function(result) {
                let userID = result.get("UserID");
                let userEmail = result.get("User Email");
                // Push each processing promise to the array
                processingPromises.push(ProcessUserData(userID, userEmail));
            });
            getNextPage();
        }, async function done(error) {
            if (error) {
                console.error(error);
                reject("Data retrieval failed");
            } else {
                // Wait for all processing to complete
                try {
                    await Promise.all(processingPromises);
                    resolve(1);
                } catch (processingError) {
                    reject("Data processing failed");
                }
            }
        });
    });
}

This collects all the processing promises and waits for them to finish before resolving. Works perfectly.

3

your ProcessUserData might be async but u gotta await it. change forEach to async and await ProcessUserData or collect processing promises in an array and use Promise.all() before resolving. done callback is for fetching pages, not for when processing is done.