Struggling to input text and choose dropdown options in Puppeteer for web scraping

Other Questions Puppeteer
1

I’m doing a web scraping project using Puppeteer and JavaScript. I’m trying to get info from a property website. The tricky part is filling out a form, especially the ‘Comuna’ field.

I’ve tried different ways to fill it:

// Attempt 1
await page.focus('input[data-ng-model="communeName"]');
await page.keyboard.type('SAN', { delay: 50 });

// Attempt 2
page.type('input[data-ng-model="communeName"]', 'SANTIAGO', { delay: 50 });

// Attempt 3
await page.evaluate(() => {
  const communeInput = document.querySelector('input[data-ng-model="communeName"]');
  communeInput.value = 'VALPARAISO';
  communeInput.dispatchEvent(new Event('input', { bubbles: true }));
});

// Attempt 4
await page.locator('input[data-ng-model="communeName"]').fill(commune);

But nothing works. The other fields like ‘Block’ and ‘Plot’ were easy to fill. Any ideas on how to fix this? Thanks!

2

I’ve faced similar challenges with Puppeteer, especially when dealing with dynamic forms. From my experience, the issue might be related to how the website handles input events. Have you tried simulating user interaction more closely? Here’s an approach that’s worked for me:

await page.click('input[data-ng-model="communeName"]');
await page.keyboard.type('SAN');
await page.waitForTimeout(1000); // Give the dropdown time to populate
await page.keyboard.press('ArrowDown');
await page.keyboard.press('Enter');

This method mimics how a user would interact with the field. It clicks the input, types part of the name, waits for the dropdown to appear, then selects an option. The timeout might need adjusting based on the site’s responsiveness.

If that doesn’t work, you might need to investigate if the site uses any custom event listeners or if there’s any AJAX call triggered on input. Using the browser’s developer tools to monitor network activity and JavaScript events could provide more insights.

3

Have you considered using Puppeteer’s waitForSelector method before interacting with the input? This ensures the element is fully loaded and ready for interaction. Here’s an approach that’s worked well in my projects:

await page.waitForSelector('input[data-ng-model="communeName"]');
await page.click('input[data-ng-model="communeName"]');
await page.keyboard.type('SAN');
await page.waitForSelector('.dropdown-option', { visible: true });
await page.keyboard.press('ArrowDown');
await page.keyboard.press('Enter');

This method waits for the input to be available, types the text, then waits for the dropdown options to appear before selecting one. You might need to adjust the selector for the dropdown options based on the site’s structure.

If this doesn’t work, the site might be using a complex framework or custom events. In that case, you may need to inject JavaScript directly into the page to trigger the appropriate events or update the underlying data model.

4

hey, have u tried using page.evaluate() to directly set the value and trigger the change event? something like this might work:

await page.evaluate(() => {
  const input = document.querySelector('input[data-ng-model="communeName"]');
  input.value = 'SAN';
  input.dispatchEvent(new Event('change', {bubbles: true}));
});

this bypasses any weird event listeners the site might have. worth a shot!