I notice that MAPPO is supportted in 0.11.0

[394262597]

Hi, I notice that MAPPO is supportted in 0.11.0, and I'm really eager for using MAPPO algorithm in NVIDIA Isaac Sim, but this work may have not shown for us. Could you please tell me what time may I use MAPPO? In addition, If I create MAPPO class in skrl.agents.torch by myself, is it possible to work?
Very thanks!

[Toni-SM]

Hi @394262597

Multi-agent reinforcement learning is one of the feature I am working for the next (mayor) release.

However, you can try it with skrl by changing to or cloning the multi-agent branch.
Available algorithms are MAPPO and IPPO, at the moment.

In the next link you can find more information and an example for the Bi-Dexhands ShadowHand Over environment (created on top of Isaac Gym preview 4).

#51 (reply in thread)

It should be straightforward to adapt for MAPPO and Omniverse Isaac Gym

[394262597]

Hi @Toni-SM, I'm honored to receive your reply, and I will try the toturial link above later.
I'm sure it should be straightforward to adapt for MAPPO and Omniverse Isaac Gym, for there are successful examples in https://github.com/PKU-MARL/DexterousHands#Tasks. Maybe... I'm looking forward that I want to use MAPPO in Omniverse Isaac Sim just like https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/tutorial_gym_new_rl_example.html. Maybe... it should update some settings to run it successfully? My confusion is that both Omniverse Isaac Sim and Omniverse Isaac Gym belong to Vec_env type, so using the same example directly on isaac sim should also work, and I want to have a try.

Whether my idea works, please get back to me. I would appreciate it.

Thanks again!!!

[Toni-SM]

Hi @394262597

A question: Do you want to train in one environment or in massive parallel environments?

In the case of massive parallel environments, using the Isaac Sim custom RL example (from the link you indicated) is not a good idea.
It is better to move to Omniverse Isaac Gym RL frameworks as indicated in this post: https://forums.developer.nvidia.com/t/unable-to-train-multi-environment-robot/236048/9?u=toni.sm

[394262597]

Hi @Toni-SM

Thank you very much for your patient reply. You're right, I want to train in massive parallel environments, and it's not a good idea in custom RL example. But my project begins at Isaac Sim custom RL example just because the example can use stable-baselines3 to set different RL algorithms like PPO, TD3, DDPG, so I have to accept to train my task in one environment.

After receiving your advice, I'm gonna try to move my code to Omniverse Isaac Gym RL frameworks to train in massive parallel environments. However, I'm still confused if I want to train in one environment, how to edit some settings to run MAPPO in skrl successfully?
In addition, after reading the link, I found some missing configuration, should I need to complete it?

Be grateful.

[Toni-SM]

Hi @394262597

The training/evaluation of multi-agent RL algorithms using skrl requires the environment (wrapped environment) to have a specific interface.
The wrapped environment interface follows the Famama PettingZoo API as show in https://skrl.readthedocs.io/en/multi-agent/api/envs/multi_agents_wrapping.html

In your case, it is necessary to program the wrapper (inheriting from skrl's MultiAgentEnvWrapper base class)...
or (better!?) design your environment to follow the Bi-DexHands interface, then you can just use the skrl's Bi-DexHands wrapper

In the second case, your Omniverse Isaac Gym environment must have the following properties:

• num_envs: int
• num_agents: int
• observation_space: list of gym/gymnasium spaces (one for each agent)
• action_space: list of gym/gymnasium spaces (one for each agent)
• share_observation_space: gym/gymnasium space (specification for a single agent, although all agents must have the same space)

Also, the observation, shared observation, reward, done and action tensors must have the following shape:
(num_envs, num_agents, DATA_SIZE), where DATA_SIZE depends on the nature of the data.

DATA_SIZE e.g: reward: 1, done: 1, observation: number of observation, etc....

I hope it will be useful :)

[394262597]

Thank you very much for your reply. I have noticed that necessary multi-agent environment must be used in omni isaac gym, and I plan to modify it bit by bit. I think it's a big project and hopefully I can cope with him.

In addition, if there is any other development, I will inform you as soon as possible.