Latest Mujoco Unity Broke MJ Geom

[fayezsalka]

Hi!

With latest Mujoco Unity plug in, all mesh based MJ Geom explodes when enetring play mode in Unity, with all parts scattered and in random orientations in the scene [Update: This looks like a new regression in latest versions of Mujoco, reverting to older releases fixes the issue].

In Addition, most mujoco_menagerie imports seem to fail. With common error:

OverflowException
Mujoco.StlMeshParser.ParseBinary (System.Byte[] stlFileContents, UnityEngine.Vector3 scale) (at ./Packages/mujoco/Editor/Importer/StlMeshParser.cs:76)
Mujoco.MjImporterWithAssets.CopyMeshAndRescale (System.String sourceFilePath, System.String targetFilePath, UnityEngine.Vector3 scale) (at ./Packages/mujoco/Editor/Importer/MjImporterWithAssets.cs:182)
Mujoco.MjImporterWithAssets.ParseMesh (System.Xml.XmlElement parentNode) (at ./Packages/mujoco/Editor/Importer/MjImporterWithAssets.cs:161)
Mujoco.MjImporterWithAssets.ParseAssets (System.Xml.XmlElement parentNode) (at ./Packages/mujoco/Editor/Importer/MjImporterWithAssets.cs:126)
Mujoco.MjImporterWithAssets.ParseRoot (UnityEngine.GameObject parentObject, System.Xml.XmlElement parentNode) (at ./Packages/mujoco/Editor/Importer/MjImporterWithAssets.cs:89)
Mujoco.MjcfImporter.ImportXml (System.Xml.XmlDocument mjcfXml, System.String name) (at ./Packages/mujoco/Runtime/Importer/MjcfImporter.cs:100)
UnityEngine.Debug:LogException(Exception)
Mujoco.MjcfImporter:ImportXml(XmlDocument, String) (at ./Packages/mujoco/Runtime/Importer/MjcfImporter.cs:104)
Mujoco.MjImporterWithAssets:ImportString(String, String, String) (at ./Packages/mujoco/Editor/Importer/MjImporterWithAssets.cs:75)
Mujoco.MjImporterWithAssets:ImportFile(String) (at ./Packages/mujoco/Editor/Importer/MjImporterWithAssets.cs:60)
Mujoco.MjImporterEditorWindow:Apply() (at ./Packages/mujoco/Editor/Importer/MjImporterEditorWindow.cs:28)

Steps for reproduction

Import any scene from mujoco_menagerie into Unity

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[fayezsalka]

Investigating more into this: Import issue seems to happen anytime mujoco unity tries to load a text mesh (.obj ASCII), the number of vertices is red incorrectly by the STL loader.

The other more significant issue, any "mesh" based obj geom is completly broken, going into play mode all meshes go into completly incorrect positions, rendering the current Unity plug in unusable. Not sure if this is recent regression / new issue in latest commits. Will try to revert to older commits to see if this fixes the issue.

[fayezsalka]

Update #2: reverting to older version of Mujoco and the Unity plug in fixes the mesh position / orientation issues. This seems to be nasty regression with all mesh based geom with the latest versions Mujoco.

The Overflow exception during import however seems to be an original bug, older version still have the issue. For now the easiest fix is to convert all text based mesh assets into binary before importing mujoco scene into unity

[fayezsalka]

The MJ Geom regression was intorduced AFTER latest release of Mujoco (latest commits)
Downloading source from 3.2.5 works correctly

[yuvaltassa]

I don't think we ever supported ASCII STL files?

The last few messages were a bit confusing, could you please clarify/repreat, what the two issues are, which one is new and which one is old, and how to reproduce them?

[Balint-H]

Hello! I did some early tests with importing mesh-based models with 3.2.5 and they seem to work (the MPL hand is my go-to test scene).

One possible reason the type of issue you are describing (meshes being in the wrong spot and orientation) can arise is if there's a mismatch between package version and mujoco binary version. Could you confirm that those match for you? If they do and the problem still persists I'll investigate deeper. In the mean time I'll also try loading some menagerie models (starting with ALOHA).

Update: The overflow issue is due to a lack of support for obj files for the Unity plugin, which most menagerie models use to differing degrees (e.g., the leap hand does not). Obj file support is definitely feasible and I started as well, but I remember someone from DeepMind was also working on it, so I stopped. I'll ask if its possible to release that feature for the public version, if it is ready.

Immediately the only solution I can suggest is batch converting obj files to stl with software such as Blender (which is how I have been using models that had obj meshes), and updating the XML files to point to them.

On the short term (today) I'll send a PR that at least detects if an OBJ file is being loaded and gives a more useful exception instead of breaking when attempting to read it as an STL.

On the long term there should be support OBJ files. In fact, Unity has a lot more default functionality for OBJ files than STL. Adding this feature is second on my list after adding preliminary support to the elasticity plugin.

[fayezsalka]

Thanks! I figured to batch convert all ASCII .obj to binary .stl using trimesh python library, and that works as workaround for now. Small issues with the converted STL meshes in unity having soft normals for all faces but I was able to work aound that too.

Regarding the MJ Geom parts being scattered all around when entering play mode, it could be because I cloned the mujoco repo (to get the latest Unity package) from the very latest commit, but then got the windows binary from the 3.2.5 release (which is couple of commits behind. So there could be mismatched like you said.

[yuvaltassa]

Good to know. Closing.