Available input actions in OpenAI Universe environment

[FlyingEagle]

I’m working with OpenAI Universe and need to control my agent’s interactions with the environment. Specifically, I want to simulate keyboard presses for keys like ‘w’, ‘space’, and other characters, as well as mouse clicks at specific coordinates.

I’ve discovered that keyboard inputs can be handled using a format like ('KeyEvent', 'W', True) where the third parameter indicates key press or release. For mouse interactions, the pattern appears to be ('PointerEvent', x_coord, y_coord, button_mask) where you specify the screen coordinates and which mouse button to use.

Can someone provide more comprehensive documentation or examples of the complete range of input events supported? I’m particularly interested in understanding all the available key codes and mouse button configurations that can be used with Universe environments.

[Isaac_Cosmos]

Yeah, Universe’s docs are pretty terrible. The KeyEvent format you mentioned works, but watch out for the key naming - regular letters use uppercase, special keys like ‘Return’, ‘BackSpace’, ‘Tab’ follow X11 keysym names.

For mouse stuff: button_mask uses 1 for left click, 4 for right, 2 for middle. You can add them together for multiple buttons. Coordinates match the VNC display resolution.

Couple things that bit me - some environments hate fast key sequences without delays between events. Also, reset the environment first or your inputs might just get silently dropped.

[sofiap]

From my experience with Universe agents, everyone misses this key thing - timing between input events. The environment handles inputs asynchronously, so if you spam commands too fast, stuff breaks. I started adding a tiny delay (50-100ms) between KeyEvent calls and it fixed most reliability issues. For PointerEvent coordinates, different environments have different screen resolutions. I always check the observation space dimensions first so my mouse coordinates don’t go out of bounds. Also, some environments need you to move the mouse first, then click - send separate PointerEvent commands instead of trying to do both at once. Pro debugging tip that saved me tons of time: turn on the VNC viewer to actually see if your inputs are working before you assume your code’s broken.

[CreatingStone]

The Problem:

You’re trying to send keyboard and mouse inputs to an OpenAI Universe environment, but you need a comprehensive guide on the supported events and their parameters. You’ve found partial information about KeyEvent and PointerEvent, but lack details on key codes, mouse button configurations, and potential pitfalls.

Step-by-Step Guide:

Step 1: Understand Input Events and Parameters

OpenAI Universe uses custom input events. The core types are:

• KeyEvent: Simulates keyboard presses and releases. The format is ('KeyEvent', key_code, is_press).

  • key_code: Uses uppercase for regular letters (e.g., ‘W’, ‘A’, ‘S’, ‘D’). For special keys, use X11 keysym names (e.g., ‘Return’, ‘BackSpace’, ‘Tab’). See the vnc_event.py file in the Universe GitHub repository (linked below) for a complete list.
  • is_press: True for a key press, False for a key release.

• PointerEvent: Simulates mouse actions. The format is ('PointerEvent', x, y, button_mask).

  • x, y: Coordinates relative to the environment’s VNC display resolution. Always check the observation space dimensions to avoid out-of-bounds errors.
  • button_mask: Use bitwise addition. 1 for left click, 2 for middle click, 4 for right click. For example, 7 (1 + 2 + 4) simulates clicking all three buttons simultaneously.

Step 2: Consult the vnc_event.py File

The vnc_event.py file (located in the OpenAI Universe GitHub repository) provides a definitive list of supported key codes and mouse button configurations. This is your primary reference for accurate input event parameters. A direct link to this file will be incredibly useful and should be included in the original post. (This step requires finding and linking the GitHub file. I cannot access external websites.)

Step 3: Incorporate Timing and Sequencing

Universe handles inputs asynchronously. Avoid sending events too rapidly. Introduce small delays (50-100ms) between consecutive KeyEvent calls to enhance reliability. In some environments, you need to send separate PointerEvent commands for mouse movement and clicks, rather than attempting both simultaneously.

Step 4: Visual Verification (VNC Viewer)

Use the VNC viewer to visually inspect your agent’s actions. This immediately confirms whether your input events are being correctly interpreted by the environment or if the problem lies in your code’s logic.

Step 5: Check Environment Resolution

Before sending PointerEvent, obtain the environment’s screen resolution from the observation space. This ensures that your mouse coordinates remain within the environment’s bounds, preventing unexpected behavior.

Step 6: Explore Examples (mini_pacman.py)

The mini_pacman.py example (also part of the OpenAI Universe repository) demonstrates effective keyboard and mouse input sequences in a game environment. Reviewing this code provides practical insights into proper usage. (This step also requires finding and linking the GitHub file.)

Common Pitfalls & What to Check Next:

• Incorrect Key Codes: Double-check that you’re using the correct X11 keysym names for special keys.
• Out-of-Bounds Coordinates: Ensure your PointerEvent coordinates are within the environment’s resolution.
• Rapid Event Sending: Introduce delays between inputs to avoid overwhelming the environment.
• Asynchronous Input Handling: Remember that inputs are processed asynchronously. Don’t assume immediate effect.
• Environment-Specific Behavior: Different Universe environments may have unique quirks.

Still running into issues? Share your (sanitized) config files, the exact command you ran, and any other relevant details. The community is here to help!