Retrieving over 100 records from Airtable API: Implementing pagination

Other Questions Airtable
1

Hey everyone! I’m struggling with the Airtable API. I managed to get it working, but I can only fetch 100 records at a time. Here’s my current code:

fetch_url = 'https://api.airtable.com/v0/MY_BASE/MY_TABLE'
auth_headers = {
    'Authorization': 'Bearer MY_SECRET_KEY'
}

result = requests.get(fetch_url, headers=auth_headers)
table_data = result.json()

This works fine, but I need more than 100 rows. I’ve heard about offset and pagination, but I’m not sure how to use them with my code. Can anyone help me figure out how to get all the records? Thanks a bunch for any advice!

2

I’ve dealt with this exact issue before, and I can share what worked for me. The key is to implement a loop that uses the ‘offset’ parameter. Here’s a basic structure:

all_records = []
offset = None

while True:
    params = {'offset': offset} if offset else {}
    response = requests.get(fetch_url, headers=auth_headers, params=params)
    data = response.json()
    
    all_records.extend(data['records'])
    
    if 'offset' in data:
        offset = data['offset']
    else:
        break

print(f'Total records fetched: {len(all_records)}')

This approach keeps fetching records until there’s no more ‘offset’ in the response. It’s efficient and works well even with large datasets. Just be mindful of rate limits if you’re dealing with a massive number of records. Hope this helps!

3

yo alexj, i’ve been there! try this:

while True:
response = requests.get(fetch_url, headers=auth_headers, params={‘offset’: offset} if offset else {})
data = response.json()
table_data.extend(data[‘records’])
if ‘offset’ not in data:
break
offset = data[‘offset’]

this shud grab all ur records. good luck!

4

I’ve encountered this issue with Airtable’s API before. Here’s a solution that worked well for me:

import requests

all_records = []
offset = None
fetch_url = 'https://api.airtable.com/v0/MY_BASE/MY_TABLE'
auth_headers = {'Authorization': 'Bearer MY_SECRET_KEY'}

while True:
    params = {'offset': offset} if offset else {}
    response = requests.get(fetch_url, headers=auth_headers, params=params)
    data = response.json()
    all_records.extend(data['records'])
    if 'offset' not in data:
        break
    offset = data['offset']

print(f'Retrieved {len(all_records)} records in total.')

This approach efficiently handles pagination by utilizing the offset parameter. It continues fetching records until no more are available. Remember to handle potential API rate limits if dealing with a large dataset.