Playwright browser class functions properly in isolation but throws errors within FastAPI route handlers

Other Questions Headless Browser
1

I built a browser automation class using Playwright for my FastAPI application. When I test the class by itself, everything works perfectly. However, when I try to use the same class inside a FastAPI route, it breaks.

Here’s my browser wrapper class:

from playwright.async_api import async_playwright
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class WebScraper:
    browser_instance = None
    playwright_process = None
    
    @classmethod
    async def start_browser(cls):
        try:
            if cls.browser_instance is None:
                if cls.playwright_process is None:
                    cls.playwright_process = await async_playwright().__aenter__()
                cls.browser_instance = await cls.playwright_process.chromium.launch(headless=True)
            logger.info('Browser started successfully.')
        except Exception as e:
            logger.error(f'Browser startup failed: {e}')

    async def fetch_html(self, target_url):
        if self.browser_instance is None:
            raise RuntimeError("Browser not started. Call start_browser first.")

        ctx = await self.browser_instance.new_context()
        tab = await ctx.new_page()
        await tab.goto(target_url)
        html_content = await tab.content()
        await ctx.close()
        return html_content

    @classmethod
    async def scrape_url(cls, target_url):
        await cls.start_browser()
        scraper = cls()
        return await scraper.fetch_html(target_url)

This works fine when called directly:

async def test_function():
    result = await WebScraper.scrape_url("https://httpbin.org")
    print(result)

asyncio.run(test_function())

But fails inside FastAPI:

from fastapi import FastAPI

api = FastAPI()

@api.get("/scrape/")
async def scrape_endpoint():
    result = await WebScraper.scrape_url('https://httpbin.org')
    return result

Error message:

raise RuntimeError("Browser not started. Call start_browser first.")
RuntimeError: Browser not started. Call start_browser first.
2

this looks like a fastapi lifecycle problem. try using lifespan events to manage your browser instance better because the playwright process might be killed between requests. also, make sure your class variables are sticking around across requests since fastapi’s async handling is a bit different from regular asyncio.run().

3

Had this exact issue when I migrated my scraping service to FastAPI. FastAPI spawns the async context in a different event loop than your standalone test. Your __aenter__() creates a playwright context that gets destroyed when the request handler finishes, but your class variable keeps pointing to nothing. I fixed it by keeping the playwright instance alive at the app level instead of using class variables. Create a global playwright instance during app startup and pass it to your WebScraper constructor. Don’t try managing it as a class variable. Your async context manager pattern works fine alone, but FastAPI’s request-response cycle messes with the context lifetime.

4

FastAPI’s handling async contexts differently than standalone asyncio - that’s your problem. The playwright process gets cleaned up between requests because FastAPI’s request lifecycle doesn’t keep the async context manager state around. I ran into the same thing with database connections. When you call aenter on async_playwright(), it creates a context that gets garbage collected after the first request finishes. You’ve got to initialize the playwright process during app startup, not on first use. Move your browser initialization to FastAPI’s startup event handler and make sure you call aexit during shutdown or you’ll leak resources. Also heads up - your class variable assignment isn’t thread-safe with multiple requests, so you’ll probably want to add some locking around the browser_instance creation.