PHP script timeout when fetching streaming platform APIs - performance issues

[evelynh]

I’m having major performance problems with my PHP code that calls streaming APIs. My script keeps timing out after 30 seconds and sometimes crashes completely.

The main issue: My page takes forever to load because I’m making API calls to check if streams are online. Sometimes I get this error: Fatal error: Maximum execution time of 30 seconds exceeded

Here’s my function to check if a channel is broadcasting:

function checkOnlineStatus($channel_name, $platform) {
    if($platform == 'twitch') {
        $api_url = "https://api.twitch.tv/streams/" . $channel_name;
        $response = file_get_contents($api_url);
        $found = strpos($response, $channel_name);
        if($found) {
            return true;
        } else {
            return false;
        }
    } else if($platform == 'youtube') {
        $check_url = 'https://youtube-api.com/live-status?channel=' . $channel_name;
        $data = simplexml_load_file($check_url);
        $status = $data->stream->isActive;
        
        if ($status == "true") {
            return true;
        } else {
            return false;
        }
    }
}

And another function that gets stream details:

function getStreamInfo($channel_name, $platform) {
    $channel_name = clean_input($channel_name);
    $platform = clean_input($platform);
    
    if($platform == 'twitch') {
        $stream_info = json_decode(file_get_contents("https://api.twitch.tv/channels/$channel_name/info"), true);
        
        $info = array(
            'avatar' => $stream_info[0]['user']['profile_image'],
            'stream_title' => $stream_info[0]['description'],
            'preview' => $stream_info[0]['user']['banner_url'],
            'category' => $stream_info[0]['current_game']
        );
    } else if($platform == 'youtube') {
        $stream_info = json_decode(file_get_contents("https://youtube-api.com/channel-data?id=$channel_name"), true);
        $info = array(
            'avatar' => $stream_info[0]['thumb_small'],
            'stream_title' => $stream_info[0]['channel_title'],
            'preview' => $stream_info[0]['thumb_big'],
            'category' => $stream_info[0]['game_category']
        );
    }
    
    return $info;
}

Both functions work fine but they’re super slow. How can I make this faster? I’ve seen other websites load this kind of data almost instantly.

Any suggestions would be really helpful!

[FlyingLeaf]

Had the same timeout headaches building a multi-platform stream aggregator. Your main problem is those synchronous API calls blocking everything during page load. The code structure’s fine, but file_get_contents has zero timeout handling or error recovery. Switch to cURL with proper timeouts - I use 5 seconds for connect, 10 seconds total. Stops one slow API from nuking your whole page. Real fix though? Go two-tier. Cache API responses in a database or Redis with timestamps. Run a background process that refreshes data every 2-3 minutes. Your page just reads the cache - instant loads. For background updates, use cURL multi-handle to grab multiple streams at once instead of sequentially. Cut my update time from 45 seconds to 8 for 20 channels. Also, Twitch and YouTube have rate limits you’re probably hitting during peak times. Caching fixes this since only your background process hits the APIs.

[Claire29]

for sure! using curl is a smart move, plus caching results can save a lot of time. make the calls every few mins if you can instead of each round. and if you want to keep it fresh, consider async options for quick fetches!

[pixelPilot]

The Problem:

You’re experiencing slow page load times and timeouts because your PHP code makes synchronous API calls to check the online status and retrieve information of streaming channels. The file_get_contents function blocks execution while waiting for each API response, leading to timeouts when the API calls take longer than the 30-second execution limit. This is exacerbated by the fact that you’re making multiple sequential API calls.

Understanding the “Why” (The Root Cause):

Synchronous API calls are inherently blocking. While they’re simple to implement, they halt your script’s execution until a response is received. This is unacceptable for web applications which require fast response times. When dealing with multiple API calls, this blocking behavior leads to cumulative delays, easily exceeding the execution time limit. To achieve near-instantaneous load times, you need to decouple the API calls from the main request.

Step-by-Step Guide:

1. Implement Asynchronous API Calls with a Background Process: The core solution is to offload the API calls to a background process. This process will regularly fetch and update the streaming status and information, storing it in a database or cache (like Redis). Your main PHP script then reads data from the database, which ensures nearly instantaneous responses for user requests. This prevents blocking calls from slowing down your web server and prevents API rate limits from being hit. There are several ways to achieve this:

   • Using a task queue (e.g., RabbitMQ, Beanstalkd): Your main application adds tasks to a queue. A separate worker process consumes those tasks, making the API calls and updating the database. This provides excellent scalability and fault tolerance.
   • Using a cron job or scheduled task: Create a scheduled script that periodically runs, making API calls and updating the database. This is simpler to set up but might not be as flexible or responsive as a task queue.
   • Employing a dedicated background task runner (e.g., Supervisor, systemd): These tools help manage and monitor background processes.

2. Choose a Database or Cache: Select a suitable storage mechanism for your cached data. Redis, Memcached, or a simple database like MySQL or PostgreSQL can efficiently store the streaming data.

3. Implement Data Retrieval from the Cache: Modify your checkOnlineStatus and getStreamInfo functions to fetch data from your chosen database or cache instead of directly calling the APIs. This will involve adding database interaction logic (database queries or cache lookups) to those functions.

4. Use a suitable HTTP Client Library: Replace file_get_contents with a more robust HTTP client like cURL. cURL offers better error handling, timeout management, and features like multi-handle support for concurrent API requests in your background process. Here’s an example using cURL:

function checkOnlineStatus($channel_name, $platform) {
    // ... (database/cache lookup logic) ...
}

function getStreamInfo($channel_name, $platform) {
    // ... (database/cache lookup logic) ...
}

5. Implement Error Handling and Rate Limiting: In your background process, incorporate error handling (e.g., catching exceptions and retrying failed requests) and rate-limiting strategies (e.g., using exponential backoff) to handle API issues and avoid hitting rate limits.

Common Pitfalls & What to Check Next:

• Database/Cache Configuration: Ensure your database or cache is correctly configured and accessible to your PHP scripts. Verify connection details, credentials, and performance.
• Background Process Monitoring: If using a task queue, cron job, or background process runner, regularly monitor its status and logs to identify potential issues.
• Data Consistency: Implement mechanisms (like timestamps and cache invalidation) to ensure your data remains relatively up-to-date.
• API Rate Limits: Check the rate limits for both Twitch and YouTube APIs to avoid your background process getting throttled.

Still running into issues? Share your (sanitized) config files, the exact command you ran, and any other relevant details. The community is here to help!