Playwright synchronous API fails to capture dynamically generated HTML content

Other Questions JavaScript
1

I’m working on extracting fully rendered HTML content using Playwright’s sync API to process with BeautifulSoup later. The issue is that when I navigate to a webpage, the JavaScript-generated elements aren’t being captured properly.

When I manually open the same URL in Chrome or Firefox, all the dynamic content loads perfectly and I can see every element in the dev tools. However, with Playwright, the extracted HTML is missing most of the JavaScript-generated content.

I thought the synchronous Playwright API would handle all the waiting automatically, but it seems like the dynamic elements aren’t being processed even with explicit wait conditions.

Here’s my current approach:

with sync_playwright() as pw:
    firefox = pw.firefox
    browser_instance = firefox.launch()
    ctx = browser_instance.new_context()
    webpage = ctx.new_page()
    webpage.goto(target_url)
    webpage.wait_for_load_state()
    html_output = webpage.evaluate("() => document.documentElement.outerHTML")
    
    browser_instance.close()

The html_output variable contains raw JavaScript code instead of the rendered HTML, and elements that should be dynamically created are completely missing. What’s the proper way to ensure all dynamic content is fully rendered before extracting the HTML?

2

The synchronous API doesn’t wait for all JavaScript to finish - just the initial page load. You’ll need to wait for specific elements or network activity. Try using webpage.wait_for_selector('your-dynamic-element-selector') before grabbing the HTML, or consider using webpage.wait_for_load_state('networkidle'), which waits until there’s no network activity for 500ms. It’s common for SPAs to continue making requests after the initial load. If you have an idea of how long the content takes to render, you can add a fixed delay with webpage.wait_for_timeout(2000). For framework-specific situations, ensure you are waiting for the elements that appear post-initialization, particularly in React apps where you can focus on elements with certain data attributes that signify hydration completion.

3

you gotta wait for the JS to finish. try using webpage.wait_for_function('() => document.readyState === "complete"') or look for a specific element that shows everything’s loaded. also, check if the site has lazy loading - content might only show when you scroll, so use webpage.evaluate('window.scrollTo(0, document.body.scrollHeight)') before getting the HTML.

4

I’ve hit this exact issue scraping SPAs. The problem is wait_for_load_state() only waits for the initial DOM - not for JS frameworks to finish rendering. I combine multiple waiting strategies: first wait for a key element showing the app’s initialized, then add a short timeout for any remaining async stuff. Try webpage.wait_for_selector('[data-testid="main-content"]') or whatever selector means fully loaded, then webpage.wait_for_timeout(1000). Content sometimes loads in waves, so you might need to wait for multiple elements one by one. Also check if it’s React or Vue - they usually have specific loading indicators you can target.