How to validate HTTP status codes for website URLs in Airtable records

Other Questions Airtable
1

I’m working with an Airtable base that contains around 24000 website URLs. Many of these links have formatting issues like missing slashes or extra spaces that cause them to break. I need to find a way to check which URLs are broken so I can fix them manually.

My current approach
I’ve been using a fetch request to test each URL and get the status back. Here’s my current code:

const config = input.config();
const websiteUrl = config.websiteUrl;
let responseStatus;

try {
    const apiResponse = await fetch(websiteUrl);
    responseStatus = apiResponse.status;
} catch (err) {
    responseStatus = 'failed';
}

output.set('responseStatus', responseStatus);

Problems I’m facing

  1. Redirects aren’t being handled properly. When a URL redirects, my script returns ‘failed’ instead of following the redirect to check the final status.
  2. I only get either ‘200’ for working URLs or ‘failed’ for broken ones. I want to capture the actual HTTP status codes like 404, 500, etc. for better debugging.

Can someone help me improve this script to handle redirects and capture proper error codes?

2

Both solutions work but you’ll hit massive bottlenecks trying to manually validate 24k URLs. Been there - dealt with this exact mess last year cleaning up our marketing database.

The real issue isn’t your code, it’s handling that volume efficiently. You need batching, error handling, rate limiting, and result tracking. Building all that from scratch? Weeks of work.

I automated the whole thing instead. Built a workflow that pulls URLs from Airtable in batches, validates them with retry logic, handles edge cases (redirects, timeouts, different errors), and writes results back automatically.

Runs overnight, processes everything hands-off. No manual result copying, no browser timeouts, no rate limit headaches.

For redirects specifically - capture both final status AND the redirect chain. URLs often redirect multiple times before failing, which shows you exactly what broke.

Don’t build this from scratch. Latenode handles the complexity so you can focus on fixing the actual broken URLs instead of wrestling with code.

3

Quick tip that saved me tons of headaches during a client migration - add user agent headers. Seriously.

Lots of sites block requests without proper user agents and return 403s or just hang. You might be marking valid URLs as broken because of this.

const config = input.config();
const websiteUrl = config.websiteUrl;
let responseStatus;

try {
    const apiResponse = await fetch(websiteUrl, {
        method: 'HEAD',
        redirect: 'follow',
        headers: {
            'User-Agent': 'Mozilla/5.0 (compatible; URL-Validator/1.0)'
        },
        signal: AbortSignal.timeout(8000)
    });
    responseStatus = apiResponse.status;
} catch (err) {
    responseStatus = err.name === 'AbortError' ? 'timeout' : 'failed';
}

output.set('responseStatus', responseStatus);

Also watch out for URLs with weird characters or spaces. Run them through encodeURI() first. Learned this the hard way when half my “broken” URLs were just encoding issues.

For 24k URLs, chunk into batches of 50-100 with delays between them. Most servers will rate limit or ban you otherwise.

4

set a timeout too - some urls hang forever. i use signal: AbortSignal.timeout(5000) in fetch options. watch out for rate limiting though. 24k requests fired at once will get you blocked. add delays between batches.

5

Use the Response object properties to get more details about failures. Even when fetch succeeds but returns error codes, you still get useful data like response.ok, response.redirected, and response.url for where it ended up.

const config = input.config();
const websiteUrl = config.websiteUrl;
let result = {};

try {
    const apiResponse = await fetch(websiteUrl, {
        redirect: 'follow',
        signal: AbortSignal.timeout(10000)
    });
    
    result.status = apiResponse.status;
    result.redirected = apiResponse.redirected;
    result.finalUrl = apiResponse.url;
    result.ok = apiResponse.ok;
} catch (err) {
    result.status = err.name === 'AbortError' ? 'timeout' : 'network_failed';
    result.error = err.message;
}

output.set('validationResult', JSON.stringify(result));

This gives you everything - whether redirects happened, where they landed, and tells different failure types apart. The timeout stops requests from hanging forever.

6

I hit the same issues doing bulk URL validation. Your current approach has a problem - fetch throws exceptions for network errors but not HTTP error codes like 404 or 500. You’ve got to separate network failures from HTTP status responses.

Here’s what worked for me:

const config = input.config();
const websiteUrl = config.websiteUrl;
let responseStatus;

try {
    const apiResponse = await fetch(websiteUrl, {
        method: 'HEAD',
        redirect: 'follow'
    });
    responseStatus = apiResponse.status;
} catch (err) {
    if (err.name === 'TypeError') {
        responseStatus = 'network_error';
    } else {
        responseStatus = 'timeout_error';
    }
}

output.set('responseStatus', responseStatus);

Use HEAD requests instead of GET - much faster since you’re just checking status codes. The redirect: ‘follow’ handles redirects automatically. You’ll get real HTTP codes like 404, 403, 500 instead of just ‘failed’, and network errors get proper labels.