What's the best way to handle looped navigation with a headless browser in Node.js?

Other Questions Headless Browser
1

Hey everyone! I’m working on a project where I need to automate some web tasks using Node.js. I’ve got to load a page, log in, click around, count stuff, and input text. I’ve played with horseman.js and other tools like jsdom and cheerio, but I’m hitting a wall.

My main issue is that I want to log in just once, then do a bunch of repeated actions in a loop. But I can’t figure out how to make this work with horseman.js. Here’s a rough idea of what I’m trying to do:

function doLogin(user, pass) {
  let browser = new HeadlessBrowser();
  browser.visit(siteUrl)
    .fillIn('username', user)
    .fillIn('password', pass)
    .pressButton('Submit')
    .waitForNavigation();
  return browser;
}

let myBrowser = doLogin('myuser', 'mypass');

while (someCondition) {
  myBrowser.doSomeStuff();
  // More actions here
}

Does anyone know if this is possible with horseman.js or if there’s another headless browser tool that would work better for this kind of setup? I’m open to suggestions! Thanks in advance for any help!

2

I’ve faced similar challenges with automated web tasks, and I found Puppeteer to be a game-changer for this kind of scenario. It’s a Node library developed by Google that gives you full control over Chrome or Chromium.

With Puppeteer, you can easily maintain a single browser instance across multiple operations. Here’s a rough outline of how you could approach your task:

const puppeteer = require('puppeteer');

async function run() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await login(page);

  while (someCondition) {
    await doSomeStuff(page);
    // More actions
  }

  await browser.close();
}

async function login(page) {
  await page.goto(siteUrl);
  await page.type('#username', 'myuser');
  await page.type('#password', 'mypass');
  await page.click('#submit');
  await page.waitForNavigation();
}

run();

This structure allows you to keep the same page instance throughout your operations, maintaining your logged-in state. Puppeteer’s API is intuitive and well-documented, making it easier to handle complex navigation scenarios.

3

I’ve had success using Nightmare.js for similar tasks. It’s lightweight and designed specifically for automation in Node.js environments. Here’s a basic approach:

const Nightmare = require('nightmare');
const nightmare = Nightmare({ show: false });

nightmare
  .goto('http://example.com')
  .type('#username', 'myuser')
  .type('#password', 'mypass')
  .click('#submit')
  .wait('.dashboard')
  .then(() => {
    // Your logged-in session is now active
    return performActions(nightmare);
  })
  .catch(error => {
    console.error('Login failed:', error);
  });

function performActions(browser) {
  // Loop through your actions here
  return browser
    .click('.some-button')
    .wait('.result')
    // More actions...
    .evaluate(() => document.querySelector('.count').textContent)
    .then(count => {
      console.log('Count:', count);
      // Continue with more actions or end the session
    });
}

This approach maintains a single browser instance throughout your operations, preserving the login state. Nightmare’s API is straightforward and well-suited for scripting repetitive tasks.

4

have u tried using Selenium WebDriver? it’s pretty good for this kinda stuff. you can keep the browser session open and do multiple actions. here’s a basic example:

const webdriver = require('selenium-webdriver');
let driver = new webdriver.Builder().forBrowser('chrome').build();

driver.get('http://example.com');
driver.findElement(By.id('username')).sendKeys('user');
// do more stuff in a loop

hope this helps!