Playwright Python automation works fine locally but encounters timeout issues when deployed to AWS Lambda

Other Questions Headless Browser
1

Background

I built a web automation script using Playwright in Python that runs perfectly on my local development environment. However, when I try to run the same code in AWS Lambda, it keeps timing out and fails to complete the automation tasks.

Code Sample

async def execute_automation(self) -> None:
    async with async_playwright() as pw:
        status = "pending"
        record_id = "none"
        
        browser_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
        
        automation_logger = self.log_service.create_logger(BankPortal.ID)
        chrome_browser = await pw.chromium.launch(headless=True, args=["--no-sandbox"])
        load_state = "networkidle"
        
        geo_service = Nominatim(user_agent="AutomationBot")
        mumbai_location = geo_service.geocode("Mumbai, Maharashtra")
        
        print("Location latitude: ", mumbai_location.latitude)
        print("Location longitude: ", mumbai_location.longitude)
        
        browser_context = await chrome_browser.new_context(
            ignore_https_errors=True,
            user_agent=browser_agent,
            viewport={"width": 1366, "height": 768},
            geolocation={"longitude": mumbai_location.longitude, "latitude": mumbai_location.latitude},
            permissions=["geolocation"]
        )
        
        automation_logger.debug("Browser context initialized...")
        web_page = await browser_context.new_page()
        web_page.set_default_timeout(180000)
        automation_logger.debug("New page instance created...")
        
        automation_logger.debug("Opening portal login page...")
        await web_page.goto(url=self._config["portal_url"], wait_until="load")
        
        print(f'Target URL = {self._config["portal_url"]}')
        
        automation_logger.info("Portal page loaded successfully")
        automation_logger.debug("Entering login credentials...")
        
        await sleep(8)
        automation_logger.debug("Wait period completed")
        
        automation_logger.debug("Locating username input field")
        await web_page.locator('[id="username"]').click()
        
        automation_logger.debug("Username field clicked")
        await web_page.locator('[id="username"]').fill(value=str(self._config["username"]))
        
        automation_logger.debug("Username entered successfully")
        await web_page.wait_for_load_state('domcontentloaded')
        await web_page.locator('[class="btn-primary btndisable mat-button mat-button-base ng-star-inserted"]').click()
        automation_logger.debug("OTP request button pressed")
        automation_logger.debug("OTP request completed successfully...")

Setup Details

I’m using Docker containers for Lambda deployment with custom layers and middleware. This setup works great for other automation scripts on my local Linux machine. The main difference is that this particular script requires geolocation data to function properly in headless mode, and this seems to be where Lambda deployment fails.

Specific Questions

  1. What are the main differences between running Playwright locally versus in AWS Lambda environment?
  2. How can I debug and fix the timeout problems in Lambda?
  3. Are there special settings needed for Playwright in serverless environments?

Technical Specs

  • Lambda Runtime: Python 3.9
  • Playwright Version: 1.35.0
  • AWS Region: ap-south-1

Any help would be greatly appreciated!

2

Had the same timeout headaches with Playwright on Lambda last year. It’s usually cold starts plus browser startup eating your time budget. Chromium takes forever to launch in Lambda vs running locally. Here’s what fixed it for me: bump Lambda timeout to 5+ minutes and throw more memory at it (I went with 3008 MB). Also, ditch that Nominatim geolocation call during Lambda runs - external APIs during init will kill your timeout. Just hardcode Mumbai’s coordinates or grab them beforehand. Add --disable-dev-shm-usage to your chromium args. Lambda’s /dev/shm is capped at 64MB and browsers hate that. The --single-process flag helped too since Lambda’s pretty memory-constrained. If this runs regularly, consider provisioned concurrency. Eliminates cold starts completely and makes timing way more predictable.

3

Memory limits are killing your browser process. Lambda’s default 128MB won’t work with Chromium - bump it to at least 1GB for Playwright. That geolocation service call is adding latency too, just hardcode Mumbai’s coordinates. Add --disable-background-timer-throttling and --disable-backgrounding-occluded-windows to your Chrome args - helps with Lambda’s execution environment.

4

Lambda has way stricter resource limits than your local machine. That sleep(8) in your code is just wasting timeout seconds for nothing. Use proper wait conditions like wait_for_selector or wait_for_load_state instead. Your 180 second page timeout is too aggressive for Lambda’s slower performance. Drop it to 30-60 seconds and handle failures better. The networkidle load state waits for all network activity to stop, which can hang forever in Lambda. Switch to domcontentloaded. That geolocation API call adds latency every single time. Cache those coordinates or stick them in environment variables. If you’re using VPC, check your Lambda’s config - NAT gateway routing slows down external requests big time. Watch your CloudWatch logs closely to see exactly where it’s timing out.