Hotel booking site price extraction fails with Puppeteer automation

Other Questions Puppeteer
1

I’m having trouble with my web scraping script that extracts hotel prices from booking websites. The main issue is that my date selection function selectBookingDates stopped working properly, which means the price elements never load on the page.

My script was working fine before but now the date picker interaction seems broken. I need help fixing the date selection logic and also want to properly use the year constants I defined.

Here’s my current code:

const puppeteer = require("puppeteer");

const config = {};
config.CHECKIN_MONTH = 'Jan'
config.CHECKIN_DAY = '28'
config.CHECKIN_YEAR = '2024'

config.CHECKOUT_MONTH = 'Feb'
config.CHECKOUT_DAY = '3'
config.CHECKOUT_YEAR = '2024'

const targetUrl = "https://www.hilton.com/en/";
const userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36";

let browserInstance;
(async () => {
  browserInstance = await puppeteer.launch({
    headless: false
  });
  const [currentPage] = await browserInstance.pages();
  await currentPage.setUserAgent(userAgent);
  await currentPage.setViewport({ width: 1065, height: 10700 });
  await currentPage.goto(targetUrl, {waitUntil: "domcontentloaded"});

  await currentPage.type('[class="form-input w-full"]', "Seattle, WA, USA");
  
  await currentPage.click('[data-testid="search-dates-button"]');
  await currentPage.waitForSelector('[data-testid="calendar-container"]');

  await selectBookingDates(currentPage, config);

  await currentPage.click('[data-testid="search-submit-button"]');

  const priceSelector = 'li[class^="border-border"] p[data-testid="priceInfo"]';
  await currentPage.waitForSelector(priceSelector);
  const hotelPrices = await currentPage.$$eval(
    priceSelector,
    elements => elements.map(element => element.textContent.trim())
  );
  console.log(hotelPrices);
})()
  .catch(error => console.error(error))
  .finally(() => browserInstance?.close());

async function selectBookingDates(page, config) {
    await page.waitForTimeout(3000);
    const navigateToMonth = async (monthName) => {
      for (let counter = 0; counter < 12; counter++) {
        const currentMonth = await page.$eval('[id^="calendar-month-"]', (element) => element.textContent);
        if (currentMonth.includes(monthName)) break;
  
        const navigationButtons = await page.$$(
          '[data-testid="calendar-container"] > [class="flex justify-between"] button:not([disabled])'
        );
        await navigationButtons[navigationButtons.length - 1].click();
      }
    };
  
    const selectDate = async (dayNumber) => {
      await page.waitForTimeout(3000);
      const availableDays = await page.$$eval(
        'button[id*=day-] span:not([class*=sr-only])',
        (dayElements) => dayElements.map((dayElement) => dayElement.textContent)
      );
      const targetDayIndex = availableDays.indexOf(`${dayNumber}`);
      await page.$$('button[id*=day-] span:not([class*=sr-only])').then((dayButtons) => dayButtons[targetDayIndex].click());
    };
  
    await navigateToMonth(config.CHECKIN_MONTH);
    await selectDate(config.CHECKIN_DAY);
    await navigateToMonth(config.CHECKOUT_MONTH);
    await selectDate(config.CHECKOUT_DAY);
  
    await page.click('[class="stroke-text"]');
}

The price extraction returns an empty array because the dates aren’t being selected properly. What changes do I need to make to fix the date selection function?

2

The main issue is your selectDate function clicks on span elements instead of the actual buttons. When you call dayButtons[targetDayIndex].click(), you’re clicking a span that doesn’t have proper click handlers. I ran into the same thing with Hilton’s calendar - you need to click the parent button, not the span. Change your selectDate function to use ‘button[id*=day-]’ as the selector and click those buttons directly. Find the right index by checking their inner span text. Also, those year constants aren’t being used anywhere in your date selection logic. The calendar navigation only handles months, so if you’re booking dates in a different year, you’ll need to add year navigation or make sure your target dates are in the current calendar view. One more thing - hotel sites often have anti-bot measures that break suddenly. Try adding longer waits between interactions and maybe randomize the timing to look more human.

3

the calendar navigation logic is broken - you’re checking for month name but hilton’s calendar might show “January 2024” format instead of just “Jan”. that final click on [class="stroke-text"] looks like a hardcoded selector that probably broke when they updated their UI. add some debug logging to see what month text you’re actually getting and check if those day buttons exist when you expect them to.

4

Your date selection has timing and selector issues. The selectDate function grabs button elements but tries clicking span elements - that’s a mismatch. You’re using $$eval for text content and $$ for clickable elements separately, which can give you different arrays if the DOM changes between calls.

I hit similar problems with booking sites. Waiting for specific elements beats fixed timeouts every time. Ditch waitForTimeout(3000) and use waitForSelector for the actual calendar elements you need. Also, those year constants aren’t doing anything - your navigation only handles months, so booking across years just fails silently.

For clicking, use one query that gets both text and clickable button together. Evaluate the buttons directly and find the right one by checking text content in the same context. Hotel sites love changing their dynamic class names too, so your price selector’s probably too specific and will break easily.