Ensuring complete data retrieval from Airtable API with Node.js using async/await

Other Questions Airtable
1

I’m struggling with fetching all records from Airtable using async/await in Node.js. My goal is to set a counter to 1 only after all records are retrieved. But the API’s 100-record limit is causing issues.

Here’s what I’ve tried:

let recordCount = 0;

async function fetchAllData() {
  recordCount = await tableDataFetcher.getAllRecords();
  console.log('Records fetched:', recordCount);
  
  if (recordCount > 0) {
    recordCount = await qualityMetrics.getAllRecords();
    console.log('Quality metrics:', recordCount);
  }
}

async function getAllRecords() {
  return new Promise((resolve, reject) => {
    let totalRecords = 0;
    
    myAirtableBase('DataTable').select().eachPage(
      (records, getNextPage) => {
        records.forEach(record => {
          let id = record.get('ID');
          let userEmail = record.get('Email');
          processRecord(id, userEmail);
          totalRecords++;
        });
        getNextPage();
      },
      err => {
        if (err) {
          reject(err);
        } else {
          resolve(totalRecords);
        }
      }
    );
  });
}

How can I make sure all records are fetched before the promise resolves? Any tips?

2

I’ve tackled this issue before, and here’s what worked for me:

Instead of using eachPage, I found it more reliable to use the ‘all’ method from Airtable’s API. It automatically handles pagination for you, which solves the 100-record limit problem. Here’s a snippet that might help:

async function getAllRecords() {
  let allRecords = [];
  await myAirtableBase('DataTable').select().all().then(records => {
    allRecords = records.map(record => ({
      id: record.get('ID'),
      email: record.get('Email')
    }));
  });
  return allRecords.length;
}

This approach ensures you get all records before the promise resolves. It’s been more consistent in my experience, especially with larger datasets. Just remember to handle potential API rate limits if you’re dealing with a massive number of records.

3

hey, i had similar issues b4. try using a recursive function with async/await. something like this:

async function getAllRecords(offset) {
  let records = await base('DataTable').select({offset}).firstPage()
  if (records.length === 0) return []
  return records.concat(await getAllRecords(records.offset))
}

this keeps fetching til there’s no more. hope it helps!

4

I’ve encountered similar challenges with Airtable’s API. One effective approach is to implement a recursive function that handles pagination internally. This method ensures all records are fetched before resolving the promise.

Here’s a streamlined version that might work for your scenario:

async function getAllRecords(offset = null) {
  const options = { pageSize: 100 };
  if (offset) options.offset = offset;

  const { records, offset: newOffset } = await myAirtableBase('DataTable').select(options).firstPage();

  const processedRecords = records.map(record => ({
    id: record.get('ID'),
    email: record.get('Email')
  }));

  if (newOffset) {
    return processedRecords.concat(await getAllRecords(newOffset));
  }

  return processedRecords;
}

This function recursively fetches all pages and combines the results. It’s efficient and ensures complete data retrieval before resolving.