I hit the same issues doing bulk URL validation. Your current approach has a problem - fetch throws exceptions for network errors but not HTTP error codes like 404 or 500. You’ve got to separate network failures from HTTP status responses.
Here’s what worked for me:
const config = input.config();
const websiteUrl = config.websiteUrl;
let responseStatus;
try {
const apiResponse = await fetch(websiteUrl, {
method: 'HEAD',
redirect: 'follow'
});
responseStatus = apiResponse.status;
} catch (err) {
if (err.name === 'TypeError') {
responseStatus = 'network_error';
} else {
responseStatus = 'timeout_error';
}
}
output.set('responseStatus', responseStatus);
Use HEAD requests instead of GET - much faster since you’re just checking status codes. The redirect: ‘follow’ handles redirects automatically. You’ll get real HTTP codes like 404, 403, 500 instead of just ‘failed’, and network errors get proper labels.