I’m working on a custom OpenAI Gym environment and facing an issue with variable action spaces. My environment has a total of 46 possible actions, but depending on the current state, only a subset (like 7 actions) might be valid at any given time.
I’ve tried looking for solutions but haven’t found a clear way to implement this. The gym documentation doesn’t seem to cover this scenario well. I’m also using a DQN agent and I’m not sure how it selects actions when the available options change.
Does anyone know the best approach to handle state-dependent action spaces? How does the agent know which actions are valid in each state? Any suggestions or examples would be really helpful.
Had the same issue building a trading bot where actions changed based on portfolio state. Here’s what worked: keep your action space at 46 but add action masking in the step function. When the agent picks an invalid action, either make it a no-op with zero reward or map it to the nearest valid action. For DQN, mask invalid actions by setting their Q-values to negative infinity before epsilon-greedy or softmax selection. The agent learns what’s typically invalid without breaking the gym interface. Also consider adding action validity to your observation space - helps the agent connect state features to available actions much better.
you can also use action_mask in the info dict from step(). most modern RL libraries like stable-baselines3 handle this out of the box. just return info={'action_mask': valid_actions_boolean_array} and the agent will only look at masked actions during training. much cleaner than manually tweaking q-values.