A little problem about MARL env in IPPO

[394262597]

Hello, I am sorry to disturb you again. I am a graduate student who consulted you before about implementing MAPPO. After watching the example you provided, "ippo_bidexhands_shadow_hand_over," I have some questions I would like to ask you:

• I have not yet understood the relationship between observation_spaces, action_spaces, and env.observation_space(agent_name) in agent = IPPO(). Is env.observation_spaces equivalent to "for _ in range(env.observation_space(agent_name))"?

• In IPPO, if possible_agents are added, are there any key changes required besides modifying corresponding models, memories, and observation_spaces? Additionally, are the same modifications required for the multiagent class?

Thanks again and looking forward your reply.

[Toni-SM]

Hi @394262597

Both observation_spaces and observation_space are defined following the PettingZoo implementation:

• observation_spaces: a dictionary of the observation spaces of every agent, keyed by name.
• observation_space: a method that returns the observation space for a specific agent.
• possible_agents: a list (of strings) of all possible agents the environment could generate.

Then env.observation_spaces is equivalent to {agent: env.observation_space(agent) for agent in env.possible_agents}

The same for the actions and shared observations

---

Regarding point 2. Could you please extend it? I can't quite understand what is the desired behavior.

[394262597]

Hi @Toni-SM
Sorry for replying to your message so late.

Thank you very much for your answer to my first question, then I looked at the PettingZoo environment code and added the above attributes to my isaac environment.

As for my second question, the trainer.train() function you wrote is mainly implemented for a single agent. If I want to expand it to multiple agents, can I change the self.agents property? Because I noticed that you have defined the number of agents in trainer.train(). After changing self.agents property, self.agents was also added to PPO, observation_spaces and possible_agents in the environment were called, and corresponding policy and value networks were generated based on the number of agents.
For the implementation of IPPO, I did not consider directly changing the PPO at first, but generating a different agent, such as agent1 = PPO(models = models_ppo1...). After that, I was trained by the trainer.train, but I tried it and it seemed impossible. I wonder if this attempt is correct, or must the above necessary parameters be added to the PPO?

Looking forward to your reply!

[Toni-SM]

A question: are you using skrl's IPPO (from multi-agent branch) or some custom implementation?

[394262597]

I want to use skrl's IPPO as shown in your link. However, in the latest release, I think I should define the MultiAgent class and IPPO class. So I can't use skrl's IPPO directly for there's no complete function definition.

[Toni-SM]

Hi @394262597

It seems to be you are not using the multi-agent branch (not released yet) that have the multi-agents implementation (IPPO and MAPPO).

You can do git checkout multi-agent if you installed skrl from a clone of their repository in editing mode...
or install it via pypi as follow: python -m pip install git+https://github.com/Toni-SM/skrl.git@multi-agent

[394262597]

Hi @Toni-SM

You are quite right, I'm not using the multi-agent branch, and that's the problem. I will follow your tutorial to install it.

Thank you very much!