How to retrieve all Airtable records beyond the 100 limit using axios requests?

Other Questions Airtable
1

I’m working with Airtable API and running into an issue where I can only get 100 records at a time. My table has way more entries than that and I need to pull all of them into my application. I’ve been using axios to make the API calls but it seems like there’s some kind of pagination happening that I’m not handling properly. Here’s my current code that only gets the first batch:

fetchAllData(){
  const self = this;
  const baseId = "**********";
  const apiToken = "**********";
  axios.get(
      `https://api.airtable.com/v0/${baseId}/Monthly%20Data?view=Grid%20View`,
      {
          headers: { Authorization: `Bearer ${apiToken}` }
      }
  ).then(function(result){
    console.log(result.data.records);
    self.dataList = result.data.records;
  }).catch(function(err){
    console.log(err)
  });
}

What’s the best way to handle this pagination and get all records from my Airtable base?

2

Had the same issue when I started with Airtable’s API. The key is that each response has an offset field when more records exist. I use a while loop that collects all records before processing. Works better than recursion, especially with network timeouts or connection hiccups. Watch out for rate limiting - Airtable caps you at 5 requests per second. Found this out the hard way when my requests got throttled on large datasets. Add a small delay between requests if you’re pulling tons of data. Also, bump the page size to 100 records per request with &pageSize=100 in your URL. Cuts down the total API calls you’ll need.

3

Yep, that 100 record limit is just how Airtable’s API works. You’ll need to use the offset parameter from the response to get everything.

I always use a recursive function that keeps calling until there’s no offset left:

fetchAllData() {
  const self = this;
  const baseId = "**********";
  const apiToken = "**********";
  let allRecords = [];

  function fetchPage(offset = null) {
    let url = `https://api.airtable.com/v0/${baseId}/Monthly%20Data?view=Grid%20View`;
    if (offset) {
      url += `&offset=${offset}`;
    }

    return axios.get(url, {
      headers: { Authorization: `Bearer ${apiToken}` }
    }).then(function(result) {
      allRecords = allRecords.concat(result.data.records);
      
      if (result.data.offset) {
        return fetchPage(result.data.offset);
      } else {
        self.dataList = allRecords;
        return allRecords;
      }
    });
  }

  return fetchPage().catch(function(err) {
    console.log(err);
  });
}

Just check for result.data.offset in each response. If it’s there, more records are waiting.

I’ve done this hundreds of times - works great. Watch out for rate limits though if you’re dealing with huge tables.

4

Hit this same issue last year migrating from Airtable. Here’s what worked for me: Add proper error handling around your pagination - network hiccups will kill the process halfway through. Catch failed requests and retry instead of restarting everything. Airtable sends records based on your table’s sort order, so keep your view sorting consistent if you care about data integrity. For large tables, add progress tracking. Just log how many records you’ve pulled so far. Makes debugging way easier when things break mid-fetch.

5

async/await is cleaner than promises here. Just loop until offset becomes undefined:

async fetchAllData() {
  let records = [];
  let offset;
  
  do {
    const response = await axios.get(url + (offset ? `&offset=${offset}` : ''), headers);
    records.push(...response.data.records);
    offset = response.data.offset;
  } while (offset);
  
  return records;
}

Way simpler than recursion and easier to debug when things break

6

This topic was automatically closed 4 days after the last reply. New replies are no longer allowed.