I built a web scraper class using Playwright for my FastAPI application. When I test the class by itself, everything works perfectly. However, when I try to use it inside a FastAPI route, it throws an error saying the browser wasn’t initialized.
My scraper class:
from playwright.async_api import async_playwright
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class WebScraper:
browser_instance = None
playwright_process = None
@classmethod
async def start_browser(cls):
try:
if cls.browser_instance is None:
if cls.playwright_process is None:
cls.playwright_process = await async_playwright().__aenter__()
cls.browser_instance = await cls.playwright_process.chromium.launch(headless=True)
logger.info('Browser started successfully.')
except Exception as error:
logger.error(f'Browser startup failed: {error}')
async def fetch_content(self, target_url):
if self.browser_instance is None:
raise RuntimeError("Browser not started. Run start_browser first.")
browser_context = await self.browser_instance.new_context()
web_page = await browser_context.new_page()
await web_page.goto(target_url)
html_content = await web_page.content()
await browser_context.close()
return html_content
@classmethod
async def scrape_url(cls, target_url):
await cls.start_browser()
scraper = cls()
return await scraper.fetch_content(target_url)
Working standalone usage:
async def test_scraper():
result = await WebScraper.scrape_url("https://httpbin.org")
print(result)
asyncio.run(test_scraper())
Failing FastAPI endpoint:
from fastapi import FastAPI
api = FastAPI()
@api.get("/scrape/")
async def scrape_endpoint():
result = await WebScraper.scrape_url('https://httpbin.org')
return result
Error message:
RuntimeError: Browser not started. Run start_browser first.
Why does this happen only in FastAPI and how can I fix it?
your class vars get wiped bc fastAPI doesn’t guarantee the same instance across requests. had the same problem with selenium a few months ago. just move the browser init into the route itself instead of relying on class storage. it’s slower but actually works.
fastAPI creates a new event loop for each request, so your class variables get reset. try using a dependency with lifespan events to manage the browser instance globally instead of relying on class vars. it’s more stable that way.
FastAPI handles async context managers differently than standalone scripts. When you call async_playwright().__aenter__() in FastAPI, the context manager doesn’t stay maintained across requests.
I hit this exact issue last year. Fix it by managing Playwright’s lifecycle with FastAPI’s lifespan events. Ditch the class variable approach and use globals initialized at startup:
Update your scraper to use the global browser instance instead of managing its own. This keeps the browser alive for your app’s entire lifecycle instead of getting garbage collected between requests.
FastAPI spins up separate worker processes, and your class variables don’t survive across different process boundaries. Each request might hit a different worker where your browser instance is None.
I debugged this exact problem for weeks. Managing browser lifecycles in distributed web apps is a nightmare - you get race conditions, memory leaks, and hanging processes when browsers don’t shut down cleanly.
The real issue? You’re trying to embed heavy browser automation inside your API. FastAPI should stay lightweight and fast.
I moved all my scraping logic to Latenode workflows. Your FastAPI endpoint just triggers the workflow and gets results back. No more browser management in your application code.
Latenode handles the browser lifecycle automatically - spins up browsers when needed, manages memory, and scales based on load. Your scraping runs isolated from your API so crashes don’t take down your whole service.
Your FastAPI route becomes dead simple: make one HTTP call to trigger the scraping workflow and return results. No more async context headaches.
Been there. FastAPI runs in a different event loop than your standalone test. Class variables don’t persist between requests the same way, so the browser instance gets lost.
Your approach has problems. Class variable storage is unreliable in web frameworks, and managing Playwright lifecycle manually sucks. You’ll get memory leaks if you don’t clean up properly.
I stopped wrestling with headless browsers in production apps years ago. Too many moving parts, resource management nightmares, scaling issues.
Now I use Latenode for web scraping workflows. Same scraping power, none of the headaches managing browser instances in your API code. Just make an HTTP call from your FastAPI route to trigger the workflow.
The workflow handles browser lifecycle automatically. You can schedule jobs, handle retries, and chain multiple scraping tasks. Way cleaner than shoving Playwright into FastAPI routes.
Your endpoint becomes simple - trigger the workflow, return results. No more worrying about event loops or browser initialization.
It’s event loop isolation. Your standalone test runs in one context, but each FastAPI request gets its own separate context. This breaks your class variable pattern for storing the browser instance. I’ve hit this exact issue before when moving scrapers to production APIs. Your browser_instance class variable shows up as None inside FastAPI routes even after you initialize it because the async context doesn’t carry over. Don’t fight with class variables here. Use FastAPI’s dependency injection instead. Create a browser dependency that initializes once and reuses the same instance across requests. You’ll get better control over the lifecycle and won’t have the browser randomly getting garbage collected. Or just initialize everything during app startup and stick it in app state. FastAPI keeps app state alive for the whole application lifecycle, unlike class variables that get weird in async web contexts.