I need help with getting a random record from my Notion database through their API. The problem is that I can only fetch 100 records per request due to API limits. My database has thousands of entries and I don’t know the total count beforehand.
Right now I have to loop through all pages from start to finish just to pick one random entry. This works okay for small databases but becomes really slow for large ones. I also run this process multiple times with a scheduled job which sometimes hits the rate limit.
Is there a smarter approach to select random entries from paginated APIs without having to fetch everything first? Looking for suggestions to make this more efficient.
Cache a bunch of record IDs locally during your first full scan, then randomly pick from that cache for later requests. Refresh it periodically or when you spot new records. Not perfect, but way more efficient than hitting the API every time you need something random.
I’ve hit this exact problem at work countless times. Both approaches work, but you’re still stuck writing and maintaining custom code for something that’s pretty standard.
I built this whole flow in Latenode instead. Set up pagination visually, add random sampling without coding reservoir algorithms, and get automatic rate limiting with built-in delays and retries.
Best part? Latenode runs this as a scheduled workflow and spreads API calls over time so you don’t hit rate limits. Need to change the logic later? Just drag boxes around instead of debugging code.
I cache metadata about page ranges so random picks run faster. The whole thing runs on its own without me watching it.
Here’s another trick that’s worked great for me: filter-based randomization. If your Notion database has date fields, timestamps, or numeric IDs, just generate random values in those ranges and use them as API filters. Pick a random date between your oldest and newest entries, fetch records created after that date, then grab the first few results. You skip pagination completely in most cases. The catch is your randomness depends on how evenly distributed that filter field is - but if your database gets regular updates, it works surprisingly well. I set up a fallback to regular pagination when the filtered query comes back empty. Cuts way down on API calls vs. methods that traverse everything.
Try reservoir sampling - it’s perfect for this. You can randomly pick records while streaming through pages without knowing how many total records exist upfront. The algorithm keeps a reservoir of k random items and replaces them with decreasing probability as you see more records. For k=1, you’d keep one record and replace it with probability 1/n for each new record n. This gives you uniform randomness and you can stop at any page once you’ve got enough samples. I used this for a similar pagination issue and cut my API calls from hundreds down to 20-30 requests while keeping perfect statistical randomness.
Had the same issue with a paginated API last year. Here’s what worked for me - I use a two-phase approach that cuts API calls by ~95%. First, I grab the total count by checking if Notion returns metadata about records or using the has_more field to estimate database size with minimal sampling. Then I generate a random page number in that range and fetch just that page, picking a random record from the 100 entries. Way better than traversing everything. I also added timing jitter for scheduled jobs so multiple processes don’t slam the rate limits at once. The randomness isn’t perfectly uniform, but it’s good enough for what I need and much faster.