Retrieving all records beyond 100 limit from Airtable using pagination

Other Questions Airtable
1

I’m having trouble getting all my data from Airtable because it only returns 100 records at a time. I tried using the regular airtable library but couldn’t get it to connect properly.

api_endpoint = 'https://api.airtable.com/v0/YOUR_BASE/YOUR_TABLE'
request_headers = {
    'Authorization': 'Bearer YOUR_TOKEN'
}

result = requests.get(api_endpoint, headers=request_headers)
data = result.json()

This code works fine but stops at 100 rows. I know I need to use offset parameter for pagination but I’m not sure how to implement it properly. Can someone help me understand how to loop through all the pages to get my complete dataset?

2

Had this exact issue last year pulling customer data from Airtable for a reporting dashboard. The offset approach works but it’s a pain to manage manually.

Here’s what I used:

all_records = []
offset = None

while True:
    params = {'pageSize': 100}
    if offset:
        params['offset'] = offset
    
    response = requests.get(api_endpoint, headers=request_headers, params=params)
    data = response.json()
    
    all_records.extend(data['records'])
    
    if 'offset' not in data:
        break
    
    offset = data['offset']

Airtable returns an offset value in the response when there are more records. No offset means you’ve hit the end.

Watch out for huge tables though - add a small delay between requests. I hit rate limits during a 10k record pull and learned this the hard way. Just throw in a time.sleep(0.1) in the loop and you’ll be fine.

3

Hit this exact pagination nightmare migrating from our old CRM to Airtable. Most people screw up because they don’t realize you keep requesting until there’s no offset field in the response.

import requests
import time

all_data = []
has_more = True
offset_token = None

while has_more:
    url_params = {}
    if offset_token:
        url_params['offset'] = offset_token
    
    response = requests.get(api_endpoint, headers=request_headers, params=url_params)
    json_data = response.json()
    
    all_data.extend(json_data.get('records', []))
    
    offset_token = json_data.get('offset')
    has_more = offset_token is not None
    
    time.sleep(0.2)  # be nice to their servers

Big gotcha - always check status codes before processing anything. Airtable throws partial failures that’ll wreck your pagination if you don’t handle errors properly.