Is there a way to fetch more than 100 records from Airtable API using pagination?

Other Questions Airtable
1

Hey everyone! I’m a beginner with the Airtable API and I’ve managed to get it working, but I’ve hit a snag. Here’s what I’ve done so far:

api_url = 'https://api.airtable.com/v0/MY_BASE/MY_TABLE'
headers = {'Authorization': 'Bearer MY_SECRET_KEY'}

response = requests.get(api_url, headers=headers)
data = response.json()

This code works fine, but it only grabs the first 100 records. I’ve heard about offset and pagination, but I’m not sure how to use them with my current setup. Can anyone help me figure out how to get all the records from my table? I’d really appreciate any tips or examples!

Thanks in advance for your help!

2

Having worked extensively with Airtable’s API, I can confirm that pagination is indeed the way to go for fetching large datasets. While the previous answers provide solid solutions, I’d like to add a few optimizations I’ve found useful in production environments.

Consider using the ‘pageSize’ parameter to control the number of records per request. This can help balance between minimizing API calls and managing memory usage:

all_records = []
offset = None
page_size = 100  # Adjust as needed

while True:
    params = {'pageSize': page_size, 'offset': offset} if offset else {'pageSize': page_size}
    response = requests.get(api_url, headers=headers, params=params)
    data = response.json()
    all_records.extend(data['records'])
    offset = data.get('offset')
    if not offset:
        break
    time.sleep(0.5)  # Adjust based on Airtable's rate limits

This approach has served me well in handling tables with tens of thousands of records efficiently.

3

I’ve been in your shoes, Alex. When I first started working with Airtable’s API, fetching large datasets was a challenge. Here’s what worked for me:

Implement a loop that continues fetching records until there’s no more data. You’ll need to use the ‘offset’ parameter and update it after each request. Also, consider adding a delay between requests to avoid hitting rate limits.

Here’s a snippet that might help:

all_records = []
offset = None
while True:
    params = {'offset': offset} if offset else {}
    response = requests.get(api_url, headers=headers, params=params)
    data = response.json()
    all_records.extend(data['records'])
    offset = data.get('offset')
    if not offset:
        break
    time.sleep(0.2)  # Add a small delay between requests

This approach has served me well for tables with thousands of records. Just remember to handle potential API errors and adjust the delay if needed.

4

yo alex, i’ve dealt with this before. u gotta use the ‘offset’ parameter in ur requests. here’s a quick example:

offset = None
while True:
    params = {'offset': offset} if offset else {}
    response = requests.get(api_url, headers=headers, params=params)
    data = response.json()
    # process data here
    offset = data.get('offset')
    if not offset:
        break

this’ll grab all ur records. good luck!