Vectorizable? Spawn Multiple Environments?

[DylanDeshler]

I have been trying to use gym.make_vec() to play games simultaneously across cpu cores but have run into many issues. It doesn't appear to work out of the box with this implementation, and each fix I make springs up many more issues.

Am I doing something wrong, does it work? Is there a better way to vectorize and spawn multiple environments?
If not, do you have plans of supporting these features? I may be willing to help, what do you think next steps would be?

My current instantiation:
envs = gym.make_vec( 'catanatron_gym:catanatron-v1', num_envs=n_envs, config={ 'map_type': 'BASE', 'vps_to_win': 10, 'enemies': [RandomPlayer(Color.RED), RandomPlayer(Color.WHITE), RandomPlayer(Color.ORANGE)], 'representation': 'mixed', 'invalid_action_reward': -10, 'normalized': False }, )

[bcollazo]

Interesting. Didn't @zarns you made a SubProcessEnv or so that was able to run many in parallel? Can you share your findings?

If nothing, @DylanDeshler I can take a look at this this weekend. Sounds like a great feature!

[DylanDeshler]

@bcollazo Thanks for responding so quickly! Happy to see this is an active repo.

Turns out the issue was because of the way I initialized the vector environment. I was following this tutorial to get it up and running.

Changing the code from what I had before (see above) to the below fixes the issues.
def make_env(): """Factory function to create an environment.""" return gym.make( 'catanatron_gym:catanatron-v1', config={ 'map_type': 'BASE', 'vps_to_win': 10, 'enemies': [RandomPlayer(Color.RED), RandomPlayer(Color.WHITE), RandomPlayer(Color.ORANGE)], 'representation': 'mixed', 'invalid_action_reward': -10, 'normalized': False, }, ) envs = gym.vector.AsyncVectorEnv([make_env for _ in range(n_envs)])

Specifically using AsyncVectorEnv rather than gym.make_vec(). It seems like gym.make_vec() instantiations a SyncVectorEnv which causes issues in expected shapes (n_ens,) vs (1,). Which is weird because the tutorial uses the two different instantiations interchangeably...

But you can mark this as resolved! Could be worth adding this issue to the readme for future ML people. It could also be adapted for the evaluation framework for large # of games, but idk if that really worth it.

[zarns]

In my pr to add MaskablePpo player, stablebaselines had subprocessVecEnv that would let multiple games play at once. Outside of stablebaselines, I'm not sure how to make that happen

#297

[bcollazo]

Ok, cool. Great to hear @DylanDeshler, and thanks for chiming in @zarns.

Ok, I'll mark as closed, and for now this issue already serves as good documentation, but if you can make a PR against the README.md in https://github.com/bcollazo/catanatron/tree/master/catanatron_gym/README.md, with a new section about "Vectorized Environments" and put the code that worked for you there, that'd be great. Could be worth pointing to @zarns 's Pull Request as well. Thanks!