Creating Custom Gaming Environment for TensorFlow/OpenAI Without gym.make() Function

Other Questions OpenAI
1

[Background]
I built a Python game that uses keyboard inputs like ‘w’ and ‘s’ for movement and ‘space’ for firing. I’d like to implement a reinforcement learning model to work with this game.

The RL code I found is designed for OpenAI’s arcade games and utilizes the command ‘gym.make(env_name)’. Unfortunately, I’m on Windows and can’t test this since gym[atari] is incompatible with my setup.

class RLAgent:
    def __init__(self, environment_id, is_training, display=False, enable_logs=True):
        self.environment = gym.make(environment_id)

[Main Question]
Is there an alternative to ‘gym.make()’ that I can use to link my RL algorithm with my custom game? Or is it necessary to create a complete gym environment wrapper?

Could using ‘pygame.surfarray.array2d()’ achieve similar functionality as ‘gym.make()’?

I am new to gym and TensorFlow, so I might not be aware of some basic concepts.

[Additional Info]
My game is structured with functions instead of classes. If I create a gym environment, do I have to convert all my functions into classes?

Here’s an outline of my main game function:

def main_game():
    global game_paused
    player_x = (screen_width * 0.08)
    player_y = (screen_height * 0.2)
    
    movement_x = 0
    movement_y = 0
    
    enemy_speed = 3
    bounce_direction = [3, 3]
    
    points = 0
    health = 3
    
    enemy_x = screen_width/1.2
    enemy_y = screen_height/1.2
    
    last_shot_time = pygame.time.get_ticks()
    enemy_shot_time = pygame.time.get_ticks()
    
    running = True
    while running:
        for action in pygame.event.get():
            if action.type == pygame.QUIT:
                pygame.quit()
                exit()
        
        enemy_x += bounce_direction[0]
        enemy_y += bounce_direction[1]
        
        if enemy_x + sprite_width > screen_width or enemy_x < 601:
            bounce_direction[0] = -bounce_direction[0]
        
        if enemy_y + sprite_height > screen_height or enemy_y < 0:
            bounce_direction[1] = -bounce_direction[1]
        
        for i in range(len(enemy_bullets)):
            enemy_bullets[i][0] -= 7
        
        for projectile in enemy_bullets:
            if projectile[0] < 0:
                enemy_bullets.remove(projectile)
        
        current_enemy_time = pygame.time.get_ticks()
        if current_enemy_time - enemy_shot_time > 450:
            enemy_shot_time = current_enemy_time
            enemy_bullets.append([enemy_x+20, enemy_y+20])
        
        input_keys = pygame.key.get_pressed()
        
        for i in range(len(player_bullets)):
            player_bullets[i][0] += 7
        
        for projectile in player_bullets:
            if projectile[0] > 1005:
                player_bullets.remove(projectile)
        
        if input_keys[pygame.K_SPACE]:
            current_shot_time = pygame.time.get_ticks()
            if current_shot_time - last_shot_time > 550:
                last_shot_time = current_shot_time
                player_bullets.append([player_x+20, player_y+20])
        
        if player_x < 0:
            player_x = 0
        if input_keys[pygame.K_a]:
            movement_x = -enemy_speed
        if player_x > 401 - sprite_width:
            player_x = 401 - sprite_width
        if input_keys[pygame.K_d]:
            movement_x = enemy_speed
        
        if input_keys[pygame.K_a] and input_keys[pygame.K_d]:
            movement_x = 0
        if not input_keys[pygame.K_a] and not input_keys[pygame.K_d]:
            movement_x = 0
        
        if player_y < 0:
            player_y = 0
        if input_keys[pygame.K_w]:
            movement_y = -enemy_speed
        if player_y > screen_height - sprite_height:
            player_y = screen_height - sprite_height
        if input_keys[pygame.K_s]:
            movement_y = enemy_speed
        
        if input_keys[pygame.K_w] and input_keys[pygame.K_s]:
            movement_y = 0
        if not input_keys[pygame.K_w] and not input_keys[pygame.K_s]:
            movement_y = 0
        
        player_x += movement_x
        player_y += movement_y
        
        game_screen.fill(blue_color)
        check_hits(points)
        show_health(health)
        pygame.draw.line(game_screen, black_color, (601, screen_height), (601, 0), 3)
        pygame.draw.line(game_screen, black_color, (401, screen_height), (401, 0), 3)
        draw_sprite(enemy_x, enemy_y)
        draw_sprite(player_x, player_y)
        
        for projectile in player_bullets:
            game_screen.blit(bullet_image, pygame.Rect(projectile[0], projectile[1], 0, 0))
            if projectile[0] > enemy_x and projectile[0] < enemy_x + sprite_width:
                if projectile[1] > enemy_y and projectile[1] < enemy_y + sprite_height:
                    player_bullets.remove(projectile)
                    points += 1
        
        for projectile in enemy_bullets:
            game_screen.blit(bullet_image, pygame.Rect(projectile[0], projectile[1], 0, 0))
            if projectile[0] + bullet_width < player_x + sprite_width and projectile[0] > player_x:
                if projectile[1] > player_y and projectile[1] < player_y + sprite_height:
                    enemy_bullets.remove(projectile)
                    health -= 1
        
        if health == 0:
            end_game()
        
        pygame.display.update()
        game_clock.tick(120)
2

You’ll need to create a custom gym environment wrapper for your game - there’s no direct alternative to gym.make() that works with arbitrary games. The pygame.surfarray.array2d() function only converts surfaces to arrays and won’t handle the environment interface that RL algorithms expect. Regarding your function-based structure, you don’t need to convert everything to classes. I kept my game logic as functions when I did something similar. The key is wrapping your main_game() function inside a gym.Env class that handles the step(), reset(), and render() methods. Your existing function can be called from within the step() method after some refactoring to accept actions as parameters instead of reading keyboard input directly. The main challenge will be modifying your input handling to accept discrete actions (0=no movement, 1=move up, 2=move down, etc.) instead of pygame key events. You’ll also need to define your observation space - typically the game screen as a numpy array using pygame.surfarray.array3d() for RGB values.

3

honestly you dont need to recreate everything from scratch. theres a simpler way - just modify your main loop to accept actions instead of keyboard inputs. create a simple wrapper that calls your main_game with predetermined moves and returns game state + reward. i did this with my breakout clone and it worked fine without full gym setup.

4

Creating a custom gym environment is actually the proper approach here, and it’s less complicated than it seems. You don’t need a complete overhaul of your existing code structure. What worked for me was creating a minimal gym.Env subclass that essentially acts as a bridge between your game and the RL agent. The core trick is to refactor your input handling section where you currently check pygame.K_w, pygame.K_s, etc. Instead of reading keyboard events, have your game loop accept an action parameter that represents the agent’s decision. Your current function-based approach is perfectly fine - I kept mine the same way. The gym wrapper just needs to implement the required methods like step() and reset(), where step() calls your modified main_game function with the agent’s action and returns the observation, reward, done flag, and info. For the observation space, you can use pygame.surfarray.array3d() to capture the screen state as a numpy array. The reward calculation should be based on your game metrics like points gained or health lost.