MRTK support for Apple VisionOs and Vision Pro

[IssueSyncBot]

Original issue opened by:

@njannink

---

With the upcomming release of Apple VisionOs what will be the vision of MRTK for supporting this and the Apple Vision Pro.

Would love some insights to be able to form our architecture roadmap.

---

ISSUE MIGRATION

Issue migrated from: microsoft/MixedRealityToolkit-Unity#11611

[IssueSyncBot]

Original comment by:

camnewnham

---

Based on the packages used some thoughts/discussion points:

• I expect MRTK3 UI to work out of the box
• I suggest MRTK3 Input package should migrate/support Unity's XR Hands package. Hopefully not a huge task, see: https://localjoost.github.io/Getting-Unity-XR-hands-to-work-on-HoloLens-and-Quest-in-a-bare-bones-project/
• There is no mention of OpenXR and instead seems to be ARFoundation (ARKit) - so some app changes required but given that minimal work is required to get an MRTK3 app running on ios/android with ARFoundation I expect this to also be mostly out of the box.

For early testing, you can apply to Unity's beta program: https://create.unity.com/spatial

Disclaimer: Not a core MRTK contributor

[IssueSyncBot]

Original comment by:

AMollis

---

Thank you @njannink , at this time we have no plans to support Apple Vision Pro. However, community contributions are welcome, and we encourage the community to add this support.

[yosun]

@njannink i think that MRTK UI components for VisionOS would at least give wider reach. (and longer extended project lifetime for those of us who have spent time with mrtk).. unity polyspatial does not have UI support for VisionOS other than a prefab button... whereas MRTK has so many of these necessary spatial computing prefabs already built

@AMollis @IssueSyncBot

[yosun]

@camnewnham

[njannink]

We are moving away from MRTK to be able to support the VisionPro and at the same time dropping support for the Hololens. After Microsoft announced that Hololens3 isn't going to happen anytime soon and they ditched MRTK it became clear that we should also leave this behind us.

[camnewnham]

It looks like things should work in immersive mode... I saw a video somewhere of the simulator running the MRTK dev templates.
Passthrough seems to be more difficult though, a big challenge being that code-based shaders are unsupported.

[faysalfdt]

Honestly it would be super helpful to have MRTK shader support fixed for Polyspatial, there's nothing essentially ready UX wise and requires a lot of rework and porting to make stuff work

[ptc-rgrasset]

We are also interested about this feature. Anyone from the community is actually trying to get that working ? (we don't want to duplicate effort here).

[yosun]

@ptc-rgrasset are you guys going to update Vuforia for VisionOS?

[ptc-rgrasset]

I can't and won't make any comment for Vuforia or PTC about VisionOS.

For now I'm mainly interested to get feedback, status, or any progress from anyone who has tried to build MRTK 3.0 (or 2.8) for VisionOS.

[yosun]

@ptc-rgrasset Ah sorry - They closed down vuforia forums and no one seems to respond to the stackoverflow, so the off chance that you could properly direct the question.

[ptc-pscheper]

Hey @yosun, if you're interested in seeing Vuforia Engine for VisionOS, feel free to share your device feedback via our GitHub issue-only repository: https://github.com/PTCInc/vuforia-engine/! 😄

[keveleigh]

I'd say I'm personally happy to help make any required changes to get MRTK up and running, within my capacity! I don't anticipate having access to a device, and the simulator seems to be Mac-only (please let me know if I'm mistaken!), so I'd be relying on everyone here to report issues and whatnot.

But PRs and specific requests are always welcomed!

[keveleigh]

Looking through some of the existing reports in this thread:

I suggest MRTK3 Input package should migrate/support Unity's XR Hands package.

support for this was added in #504, though it's not yet the default in the MRTK profile we ship.

Honestly it would be super helpful to have MRTK shader support fixed for Polyspatial

This would be a https://github.com/microsoft/MixedReality-GraphicsTools-Unity request (CC @Cameron-Micka)

given that minimal work is required to get an MRTK3 app running on ios/android with ARFoundation I expect this to also be mostly out of the box

I would assume this as well, and it's a big reason we migrated to having ARFoundation as a dependency in MRTK3 instead of re-doing it all ourselves again like we did in MRTK2. But please let us know if you run into issues!

[unnanego]

We got two devices, so I will try it out next week and post here if I don't forget

[yosun]

Hey @yosun, if you're interested in seeing Vuforia Engine for VisionOS, feel free to share your device feedback via our GitHub issue-only repository: https://github.com/PTCInc/vuforia-engine/! 😄

Posted already actually quite a while ago - PTCInc/vuforia-engine#15

Will re-post? There should be an AI to automate this already...

[jelmer3000]

Maybe interesting for the people here, I just started trying to get things to work on VisionOS without too much planning in advance.
So far, simple input works (click buttons, drag objects, play that piano) and I've done some work on the shaders: made new shader graphs for the more common shaders (from the scene below), with all properties from the original shaders (so that it's easy to swap materials). But by far not all properties' functions are implemented in the shader graphs. Also while the MRTK shaders are uber shaders that compile based on flags, I just made one bug big shader so performance might suffer eventually.

Anyway feel free to give it a try and contribute if you like.
https://github.com/jelmer3000/MixedRealityToolkit-Unity-PolySpatial

Next steps:

• Hand tracking (once we have an AVP available)
• Improve canvas shaders so that rounded corner radii are adjustable like the MRTK shader, as well as the option to color edges or create outlines. I don't know how to implement this though.
• Add PolySpatial's hover effect component as there is no other way to highlight what the user is about to select with RealityKit.

[Khalid2188]

I managed to make MRTK3 work in fully immersive mode here are steps

1- Download the sample in Apple visionOS XR Plugin and understand the Main scene
2- Disable both hands interactors under MRTK3 XR rig
3- Use gaze interactor from MRTK3 and configure it by adding correct XRI Actions following sample project
4- It should work well in Unity, Simulator and Vision Pro

mrtk3-avp.1.mp4

[Cameron-Micka]

I'm happy to fix any issues with Graphics Tools and Apple Vision Pro in immersive mode. Once I get a device, I can do some testing and sanity checks.

Supporting spatial apps is a bit harder since there are a lot more shader requirements. Eventually I'd like to have better Shader Graph support for that.

[jbienzss]

@jelmer3000 I've been taking a look at your fork and it's promising. Thank you for starting that work. I did notice that it's currently 18 commits behind MRTK/main and I was curious if you'll have time to keep your fork up to date?

I was at VisionDevCamp last week. 100 hackers, 50+ Vision Pro's all in one place. It was epic.

https://www.linkedin.com/feed/hashtag/?keywords=visiondevcamp

There was a lot of interest in getting XR Hands working. Me and two others spent most of the hack trying to get XR Hands working in Mixed Reality Unbounded (exclusive) mode. We could get ray-based interactions to work but not grab manipulations. The moment we switched to Immersive mode it started working. We even had XRI 3.0 with Near-Far interactor on the rig and we still couldn't get MR to work. I'd love to see brilliant MRTK minds like @Cameron-Micka and @keveleigh help get things working.

@Cameron-Micka, what would it take to make a case for your improved shader graph support? I notice that issue has been open for nearly 2 years. As you're probably already aware, on visionOS in Mixed Reality mode (bounded or unbounded) Unity does not actually perform the rendering. Rendering is handed off to Reality Kit and therefore custom HLSL shaders are not supported in this mode. Shaders written with Shader Graph are translated by Unity as best they can in order to fit into Reality Kit. The only way any version of Graphics Tools could possibly work in MR on AVP would be if it's written based on Shader Graph. Here's the documentation on which Shader Graph features are currently supported on visionOS:

https://docs.unity3d.com/Packages/[email protected]/manual/ShaderGraph.html

[Cameron-Micka]

@Cameron-Micka, what would it take to make a case for your improved shader graph support? I notice that issue has been open for nearly 2 years. As you're probably already aware, on visionOS in Mixed Reality mode (bounded or unbounded) Unity does not actually perform the rendering. Rendering is handed off to Reality Kit and therefore custom HLSL shaders are not supported in this mode. Shaders written with Shader Graph are translated by Unity as best they can in order to fit into Reality Kit. The only way any version of Graphics Tools could possibly work in MR on AVP would be if it's written based on Shader Graph. Here's the documentation on which Shader Graph features are currently supported on visionOS:

https://docs.unity3d.com/Packages/[email protected]/manual/ShaderGraph.html

Unfortunately since this isn't currently a requirement of my current work responsibilities, I don't see having time to do this in the short term. But it's certainly something I would like to do!

[yosun]

@Cameron-Micka what are your work responsibilities? Since MS discontinued Hololens, this is the future of spatial computing and MR/XR. There is a lot of good work that has been done here that shouldn't go wasted and could instead help kickstart a new platform. Curious if this could possibly be @ intersection of your work responsibilities (even when interpreted broadly)?

[Cameron-Micka]

@Cameron-Micka what are your work responsibilities?

Thank you for the kind works @yosun! I'm currently working on Microsoft Mesh. Which does use Graphics Tools. At the movement Mesh can be run on Microsoft Teams, Windows PC, and Quest. But if/when it runs on VisionOS (specifically spatial apps) that would build a case to start this work.