Selecting dropdown options in Puppeteer without static IDs

Other Questions Puppeteer
1

I’m working on a web scraping project using Puppeteer and Node.js. I’ve hit a snag with dropdown menus. The issue is that the IDs for the dropdown options are dynamic and change every time the page loads or an interaction occurs. This makes it tricky to select specific options.

I’ve tried a few things:

  1. Scraping the HTML to find the current ID
  2. Using XPath selectors
  3. Looking for static class names

None of these approaches have worked consistently. The dropdown looks something like this in the HTML:

<select name="homeState" id="homeState-aria-id-X">
  <option value="AL">Alabama</option>
  <option value="AK">Alaska</option>
  <!-- more options -->
</select>

Where X in the ID is a randomly generated number.

Does anyone have experience with this kind of setup? What’s the best way to select dropdown options when the IDs are dynamic? Are there other selectors or techniques I should try?

2

hey dancingbird, i’ve run into this before. try using the aria-label attribute if it’s there. like this:

await page.select(‘select[aria-label=“Select your state”]’, ‘AL’);

if that doesn’t work, you could try finding the option by text content:

await page.evaluate(() => {
const option = Array.from(document.querySelectorAll(‘option’)).find(o => o.textContent === ‘Alabama’);
option.selected = true;
});

hope this helps!

3

I’ve encountered similar issues with dynamic IDs in Puppeteer. One effective approach is to use the ‘name’ attribute, which often remains static. Try something like this:

await page.select(‘select[name=“homeState”]’, ‘AL’);

This selects the dropdown by its name and chooses the option with value ‘AL’. If that doesn’t work, you could try matching the visible text:

await page.evaluate(() => {
const select = document.querySelector(‘select[name=“homeState”]’);
const option = Array.from(select.options).find(o => o.text === ‘Alabama’);
select.value = option.value;
select.dispatchEvent(new Event(‘change’, { bubbles: true }));
});

This JavaScript snippet finds the option by its visible text and triggers a change event. Remember to adjust error handling and waiting strategies as needed for your specific use case.

4

Hey there, I’ve dealt with similar headaches in my web scraping adventures. One trick that’s worked wonders for me is using data attributes. Many devs add these for testing purposes, and they’re usually more stable than IDs.

Try looking for something like ‘data-test-id’ or ‘data-cy’ in the HTML. If you find one, you can use it like this:

await page.select(‘[data-test-id=“state-dropdown”]’, ‘AL’);

No luck with data attributes? Another approach is to use the label associated with the dropdown. Something like:

const label = await page.$(‘label:contains(“State”)’);
const dropdown = await label.$(‘select’);
await dropdown.select(‘AL’);

This finds the label containing ‘State’, then selects the dropdown within it. It’s a bit more robust against changes in the page structure. Hope this helps with your scraping project!