Browser automation script encountering CORS issues

Other Questions Puppeteer
1

Hey folks I’m having trouble with my browser automation script. It’s hitting a CORS error in most of my test runs. Here’s what I’m working with:

const automate = require('browser-bot');
const datastore = require('data-saver');

async function clickElement(selector, webpage) {
  webpage.runScript((el) => el.click(), selector);
}

async function runBot() {
  const browser = await automate.start({visible: true});
  const webpage = await browser.createPage();
  
  setTimeout(() => browser.shutdown(), 120000);
  
  await webpage.navigate('https://example.com/product/special-sneakers');
  await webpage.waitAndClick('button.buy-now');
  await webpage.pause(1000);
  
  await webpage.waitAndClick('button.size-selector');
  const sizeOption = await webpage.findElement('//ul[@class='sizes']//li[11]//button');
  await clickElement(sizeOption, webpage);
  
  await webpage.pause(1000);
  await webpage.waitAndClick('button.add-to-cart');
  await webpage.waitAndClick('a.proceed-to-checkout');
}

runBot();

The server keeps blocking the requests saying there’s no ‘Access-Control-Allow-Origin’ header. Any ideas on how to fix this? Thanks!

2

I’ve encountered similar CORS issues in my automation projects. One effective solution I found was using a proxy server to bypass these restrictions. You could set up a simple proxy using something like ‘cors-anywhere’ on your local machine or a cloud service. Then, modify your script to route requests through this proxy. It adds a bit of complexity, but it’s solved CORS problems for me consistently without needing to alter the target website’s headers or behavior. Just be mindful of the legal and ethical implications of using proxies for web scraping or automation.

3

I’ve dealt with CORS issues in my automation work before, and there’s another approach you might want to consider. Instead of trying to bypass the CORS restrictions, which can be tricky and potentially against the site’s terms of service, have you thought about using a headless browser like Puppeteer?

Puppeteer operates as a full browser, so it doesn’t typically run into CORS issues. You’d need to refactor your script a bit, but it could solve your problem more cleanly. Plus, it’s great for handling dynamic content and JavaScript-heavy sites.

Here’s a rough idea of how you might start:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();
  await page.goto('https://example.com/product/special-sneakers');
  // Use page.click() and page.waitForSelector() for your interactions
  // ...
  await browser.close();
})();

This approach has worked well for me in similar situations. Just remember to respect the website’s robots.txt and rate limits!

4

hey mia, sounds liek a pain! have you tried adding headers to ur requests? somethin like this:

webpage.setExtraHTTPHeaders({
  'Origin': 'https://example.com',
  'Referer': 'https://example.com'
});

might trick the server into thinking ur comin from the right place. worth a shot!