Intermittent JavaScript execution issues in custom WebKit browser

[emcarter]

I built a custom browser without GUI using WebKit and Qt4 with C++. The goal is to capture HTML snapshots from websites that rely heavily on JavaScript frameworks like Backbone.js.

// My approach for detecting page completion
void PageMonitor::onPageLoaded() {
    pollTimer = new QTimer();
    connect(pollTimer, SIGNAL(timeout()), this, SLOT(checkDOMChanges()));
    pollTimer->start(1000); // Check every second
}

void PageMonitor::checkDOMChanges() {
    QString currentHTML = webPage->mainFrame()->toHtml();
    if (currentHTML == previousContent) {
        pollTimer->stop();
        outputFinalResult();
    }
    previousContent = currentHTML;
}

I know there’s no perfect way to detect when JavaScript finishes executing. My solution waits for the loadFinished signal, then polls the DOM content every second to check for changes. When the content stays the same, I assume loading is complete.

The browser works great on my development machine with Apache 2.3.14 and PHP 5.4.6. However, on the production server, the same code only executes part of the JavaScript before stopping. Some sites work fine on both environments, while others only work locally.

I’ve tried increasing the timer interval but that doesn’t help. JavaScript error catching shows no issues either. The problem seems random and environment-specific.

Has anyone experienced similar issues with WebKit-based browsers in different server environments?

[scarlettturner]

This usually happens when production servers have tighter resource limits or different JavaScript contexts. I hit the same issue moving a webkit implementation to production. Turned out production had memory caps on JavaScript execution, causing frameworks to just stop running without any error messages. Check your production server’s ulimit settings and available memory - webkit eats up RAM, especially with heavy JS frameworks. Also make sure production allows the same JavaScript privileges as your dev environment. Some server configs disable certain JS APIs or cap execution time per script, which would explain why you’re seeing partial execution.

[benmoore]

Had the same issue deploying headless webkit apps across servers. It’s usually network stuff, not your code. Production has different timeouts, proxies, or firewall rules that kill resource loading halfway through. Your polling approach works, but add network timeout handlers and verify all external resources actually loaded. JS frameworks fail silently when CDN resources or API calls timeout differently between environments. Log the network requests in both setups - some resources probably aren’t reaching your production webkit instance. Also check if production has slower DNS resolution or blocks external domains your JavaScript needs.

[AdventurousHiker17]

sounds like a webkit version mismatch between your dev and prod environments. check if production’s running different qt/webkit libraries - that’ll make js behave weird even with identical code. try switching user agent strings too and see if sites respond differently.