What's the best way to locate and interact with multiple DOM elements in a headless browser?

Other Questions Headless Browser
1

I’m trying to figure out how to find and click on certain elements in a headless browser. Here’s what I want to do:

  1. Find all elements with a specific class
  2. Filter out the ones that have a particular child class
  3. Click on the remaining elements’ child buttons after a random delay

I’ve got some code that works in the browser, but I’m having trouble making it work in a headless setup like CasperJS or Horseman. Here’s a simplified version of the HTML structure I’m dealing with:

<div class="card">
  <div></div>
  <button></button>
</div>
<div class="card">
  <div class="dont-click"></div>
  <button></button>
</div>

I need to click the buttons in the cards that don’t have the “dont-click” class. Any ideas on how to make this work in a headless browser? I’m especially struggling with:

  1. Returning a list of elements from one evaluate() block to another
  2. Clicking elements using jQuery in the headless environment

Has anyone tackled something similar before? I’d really appreciate some pointers!

2

I’ve been working with headless browsers for a while now, and I’ve found that Selenium with Python offers a robust solution for this kind of task. It’s pretty versatile and works well across different browser setups.

Here’s a rough idea of how you could approach it:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import random
import time

driver = webdriver.Chrome()  # or whichever browser you're using
driver.get('your_url_here')

# Find all card elements without the 'dont-click' class
cards = driver.find_elements(By.CSS_SELECTOR, '.card:not(:has(.dont-click))')

for card in cards:
    button = card.find_element(By.TAG_NAME, 'button')
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable(button))
    time.sleep(random.uniform(0.5, 2))  # Random delay
    button.click()

driver.quit()

This approach has worked well for me in similar scenarios. It’s flexible and can be adapted to more complex situations if needed.

3

I’ve encountered similar challenges with headless browsers before. One approach that’s worked well for me is using Puppeteer. It provides a high-level API to control Chrome or Chromium over the DevTools Protocol, making it easier to interact with DOM elements.

Here’s a basic outline of how you could approach this:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('your-url');

  const buttons = await page.$$eval('.card:not(:has(.dont-click)) button', buttons => buttons.map(b => b));

  for (let button of buttons) {
    await page.evaluate((btn) => btn.click(), button);
    await page.waitForTimeout(Math.random() * 1000 + 500);
  }

  await browser.close();
})();

This script uses page.$$eval to select and return all matching buttons, then clicks each one with a random delay. It’s been reliable in my experience with headless automation tasks.

4

hey sparklinggem, sounds like a tricky situation! have u tried using querySelectorAll to grab all the cards, then filter with js? something like:

document.querySelectorAll('.card:not(:has(.dont-click)) button').forEach(btn => {
  setTimeout(() => btn.click(), Math.random() * 1000);
});

might work in headless browsers too. good luck!