VisionOS Support for VisionOSVideoComponent

[yosun]

Please support VisionOS

https://docs.unity3d.com/Packages/[email protected]/manual/VideoComponent.html?q=video

[yasirkula]

NativeGallery returns the picked video's path. How would you prefer it to behave with VisionOS?

[randalhucker]

@yasirkula To clarify, is NativeGallery confirmed to work with VisionOS? I've been fighting it all day with no luck, but would love to find out that I had just been doing something incorrectly! :)

[yasirkula]

I've heard from a couple of users that it doesn't work on VisionOS. I don't have a Vision Pro to test it myself but for the time being, we can say that it isn't supported.

[414726193]

You can use an emulator to test
Vision Pro is support PHPickerViewController.

[yasirkula]

I don't have access to a Mac workstation either and for this task, I'm relying on a fix from a volunteer (if anyone fixes the issue, please create a Pull Request).

[oOtroyOo]

lol 🤣，good question.

[randalhucker]

@yasirkula Sorry for taking so long to get back to this. I'm not very familiar with how your package worked (I only was looking for ways to get to the Gallery and stumbled here before I read that it wasn't supported), but I have a solution that prompts the gallery so that you can choose images/videos. It doesn't allow actually taking photos (VisionOS doesn't give you that permission to camera data) - but I could share that with you if it's something you think would be useful.

[yasirkula]

@randalhucker It sounds very useful for VisionOS, so I'd very much like to see your solution 👑

[RandyHucker]

@yasirkula I'll send everything in this thread when I get home tonight.

[414726193]

@yasirkula Sorry for taking so long to get back to this. I'm not very familiar with how your package worked (I only was looking for ways to get to the Gallery and stumbled here before I read that it wasn't supported), but I have a solution that prompts the gallery so that you can choose images/videos. It doesn't allow actually taking photos (VisionOS doesn't give you that permission to camera data) - but I could share that with you if it's something you think would be useful.

I really need it. Can you share it with me

[RandyHucker]

@yasirkula

Below is everything you need. All you need to do is call ImagePicker inside of a sheet or something similar: ImagePicker()

I call it like this .sheet(isPresented: $isPickerShowing) { ImagePicker() }

This does a few things... the 'selectionLimit' is how many 'items' you can pick from the gallery, and the filter is the type (i.e. photos, videos, etc.)

There are many ways to get the info out, but for me I needed one photo, so I have a class - CurrentImage - that holds the id, original extension, and the image data. You can see how I'm assigning them below.

struct ImagePicker: UIViewControllerRepresentable {
    func makeUIViewController(context: Context) -> some UIViewController {
        var config = PHPickerConfiguration(photoLibrary: .shared())
        config.selectionLimit = 1
        config.filter = .images

        let vc = PHPickerViewController(configuration: config)
        vc.delegate = context.coordinator
        return vc
    }

    func updateUIViewController(_ uiViewController: UIViewControllerType, context: Context) {}

    func makeCoordinator() -> PhotoPickerCoordinator {
        return PhotoPickerCoordinator()
    }
}

class PhotoPickerCoordinator: NSObject, PHPickerViewControllerDelegate {
    func picker(_ picker: PHPickerViewController, didFinishPicking results: [PHPickerResult]) {
        picker.dismiss(animated: true, completion: nil)

        results.forEach { result in
            result.itemProvider.loadFileRepresentation(forTypeIdentifier: UTType.image.identifier) { url, error in
                guard let url = url, error == nil else { return }

                let fileName = url.deletingPathExtension().lastPathComponent
                let fileExtension = url.pathExtension

                CurrentImage.shared
                    .with(imageName: fileName)
                    .with(imageExtension: fileExtension)

                result.itemProvider.loadObject(ofClass: UIImage.self) { image, _ in
                    guard let image = image as? UIImage else { return }

                    CurrentImage.shared.with(image: image)
                }
            }
        }
    }
}

[414726193]

@yasirkula

Below is everything you need. All you need to do is call ImagePicker inside of a sheet or something similar: ImagePicker()

I call it like this .sheet(isPresented: $isPickerShowing) { ImagePicker() }

This does a few things... the 'selectionLimit' is how many 'items' you can pick from the gallery, and the filter is the type (i.e. photos, videos, etc.)

There are many ways to get the info out, but for me I needed one photo, so I have a class - CurrentImage - that holds the id, original extension, and the image data. You can see how I'm assigning them below.

struct ImagePicker: UIViewControllerRepresentable {
    func makeUIViewController(context: Context) -> some UIViewController {
        var config = PHPickerConfiguration(photoLibrary: .shared())
        config.selectionLimit = 1
        config.filter = .images

        let vc = PHPickerViewController(configuration: config)
        vc.delegate = context.coordinator
        return vc
    }

    func updateUIViewController(_ uiViewController: UIViewControllerType, context: Context) {}

    func makeCoordinator() -> PhotoPickerCoordinator {
        return PhotoPickerCoordinator()
    }
}

class PhotoPickerCoordinator: NSObject, PHPickerViewControllerDelegate {
    func picker(_ picker: PHPickerViewController, didFinishPicking results: [PHPickerResult]) {
        picker.dismiss(animated: true, completion: nil)

        results.forEach { result in
            result.itemProvider.loadFileRepresentation(forTypeIdentifier: UTType.image.identifier) { url, error in
                guard let url = url, error == nil else { return }

                let fileName = url.deletingPathExtension().lastPathComponent
                let fileExtension = url.pathExtension

                CurrentImage.shared
                    .with(imageName: fileName)
                    .with(imageExtension: fileExtension)

                result.itemProvider.loadObject(ofClass: UIImage.self) { image, _ in
                    guard let image = image as? UIImage else { return }

                    CurrentImage.shared.with(image: image)
                }
            }
        }
    }
}

Can you share the full file, I am not an ios developer and do not know how to use this code

[yasirkula]

@RandyHucker Thank you for sharing your code 🌷 I have little-to-none Swift experience but if I understand the key parts correctly, it works similar to NativeGallery. Perhaps VisionOS only works with Swift? How are you creating and displaying a new instance of ImagePicker struct?

[randalhucker]

@yasirkula Yes, it does work very similarly. And I'm not sure if it's convertible to objective-c. I've been coding for the VPro only in Swift. And Swift makes it easy, it has @State variables which essentially force a UI-Update on every mutation. When the user clicks a button, I mutate that isPickerShowing var, and then the UI calls the .sheet method (which is like a popup)

[yasirkula]

Oh wait, I think my code doesn't present PHPickerViewController on VisionPro. @414726193 Could you remove the #if __IPHONE_OS_VERSION_MAX_ALLOWED >= 140000 and #endif lines here and change the if condition to if(YES) for testing purposes and try again:

UnityNativeGallery/Plugins/NativeGallery/iOS/NativeGallery.mm

Lines 594 to 619 in ae2ebf6

	#if __IPHONE_OS_VERSION_MAX_ALLOWED >= 140000
	if( CHECK_IOS_VERSION( @"14.0" ) )
	{
	// PHPickerViewController is used on iOS 14
	PHPickerConfiguration *config = simpleMediaPickMode ? [[PHPickerConfiguration alloc] init] : [[PHPickerConfiguration alloc] initWithPhotoLibrary:[PHPhotoLibrary sharedPhotoLibrary]];
	config.preferredAssetRepresentationMode = PHPickerConfigurationAssetRepresentationModeCurrent;
	config.selectionLimit = selectionLimit;
	pickingMultipleFiles = selectionLimit != 1;
	// mediaType is a bitmask:
	// 1: image
	// 2: video
	// 4: audio (not supported)
	if( mediaType == 1 )
	config.filter = [PHPickerFilter anyFilterMatchingSubfilters:[NSArray arrayWithObjects:[PHPickerFilter imagesFilter], [PHPickerFilter livePhotosFilter], nil]];
	else if( mediaType == 2 )
	config.filter = [PHPickerFilter videosFilter];
	else
	config.filter = [PHPickerFilter anyFilterMatchingSubfilters:[NSArray arrayWithObjects:[PHPickerFilter imagesFilter], [PHPickerFilter livePhotosFilter], [PHPickerFilter videosFilter], nil]];
	imagePickerNew = [[PHPickerViewController alloc] initWithConfiguration:config];
	imagePickerNew.delegate = (id) self;
	[UnityGetGLViewController() presentViewController:imagePickerNew animated:YES completion:^{ imagePickerState = 0; }];
	}
	else
	#endif

[414726193]

@yasirkula These errors occur during packaging debugging

[414726193]

@yasirkula
After I remove the error code, I can call this function, but it will cause this error

[yasirkula]

Can you attach the latest version of your code?

[414726193]

NativeGallery.txt
@yasirkula The code is in this file

[RandyHucker]

@yasirkula Not sure if you do something similar, but in Swift we have a 'plist' which is essentially a permissions file. I believe you might need to have the arbitrary loads enabled.

Hopefully, that helps. I'm unfamiliar with Unity's errors.

[yasirkula]

@RandyHucker Thank you. @414726193 Could you try adding it to your Info.plist? If that doesn't resolve the issue, could you put lots of NSLog statements in NativeGallery.mm (you can modify it inside Xcode for convenience) to pinpoint exactly which line crashes the app? If the stacktrace shows that line already, then that's great. I'll need to know which line does this.

[414726193]

@yasirkula

The above code runs, but the following code does not

log：

[yasirkula]

Hmm, my technical knowledge is at its limit right now. I'd recommend adding imagePickerNew.modalPresentationStyle = UIModalPresentationPageSheet; or imagePickerNew.modalPresentationStyle = UIModalPresentationFormSheet; here just to see if it works. I'm sorry for not being able to provide a definitive solution.

[yosun]

My current roundabout way is to pop up a web browser to upload a file, instead of just native file picker on visionOS

https://x.com/Yosun/status/1774521401355166148?s=20

[yasirkula]

That's smart and the interface you've created looks great IMO. May I ask how you've achieved this?

[yosun]

That's smart and the interface you've created looks great IMO. May I ask how you've achieved this?

It's kinda hacky

1. Application.OpenURL("link to your upload page?uuid=blah"); where uuid is unique tying this session to server
2. back in unity ienumerator pings server providing uuid to retrieve payload.

[yosun]

That's smart and the interface you've created looks great IMO. May I ask how you've achieved this?

Also spent a day experimenting with the antique camera obscura as a skeuomorphic interface
https://x.com/Yosun/status/1774136776082550800?s=20

[yosun]

AI stack

• object detect
• segmentation (maskrcnn could work too)
• depth mask infer z-position
• remove segmented objects and "have AI fill in the blanks"
• controlnet stylize
• segment image to 3D TripoSR - see https://github.com/yosun/ai3d for more <1 minute production usable models!

[414726193]

During this time, I finally found a solution where unity could interact with SwiftUI and wake up the gallery by calling SwiftUI
unityCode
`

delegate void CallbackDelegate(string command, int index);
// This attribute is required for methods that are going to be called from native code
// via a function pointer.
[MonoPInvokeCallback(typeof(CallbackDelegate))]
static void CallbackFromNative(string command, int index)

   [DllImport("__Internal")]
    static extern void SetNativeCallback(CallbackDelegate callback);

    [DllImport("__Internal")]
    static extern void OpenSwiftUIWindow(string name);

    [DllImport("__Internal")]
    static extern void CloseSwiftUIWindow(string name);`

swiftcode SwiftUISamplePlugin.swift

`

  typealias CallbackDelegateType = @convention(c) (UnsafePointer<CChar>,Int) -> Void

  var sCallbackDelegate: CallbackDelegateType? = nil
  
  // Declared in C# as: static extern void SetNativeCallback(CallbackDelegate callback);
  @_cdecl("SetNativeCallback")
  func setNativeCallback(_ delegate: CallbackDelegateType)
  {
      print("############ SET NATIVE CALLBACK")
      sCallbackDelegate = delegate
  }

    public func CallCSharpCallback(_ str: String,index: Int)
    {
        if (sCallbackDelegate == nil) {
            return
        }
    
        str.withCString {
            sCallbackDelegate!($0, index)
        }
    }

    // Declared in C# as: static extern void OpenSwiftUIWindow(string name);
    @_cdecl("OpenSwiftUIWindow")
    func openSwiftUIWindow(_ cname: UnsafePointer<CChar>)
    {
        let openWindow = EnvironmentValues().openWindow
    
        let name = String(cString: cname)
        print("############ OPEN WINDOW \(name)")
        openWindow(id: name)
    }
    
    // Declared in C# as: static extern void CloseSwiftUIWindow(string name);
    @_cdecl("CloseSwiftUIWindow")
    func closeSwiftUIWindow(_ cname: UnsafePointer<CChar>)
    {
        let dismissWindow = EnvironmentValues().dismissWindow
    
        let name = String(cString: cname)
        print("############ CLOSE WINDOW \(name)")
        dismissWindow(id: name)
    }

SwiftUISampleInjectedScene.swift

          struct SwiftUISampleInjectedScene {
          @SceneBuilder
          static var scene: some Scene {
              WindowGroup(id: "HelloWorld") {
                  // The sample defines a custom view, but you can also put your entire window's
                  // structure here as you can with SwiftUI.
                  HelloWorldContentView()
              }.defaultSize(width: 400.0, height: 400.0)
      
              // You can create multiple WindowGroups here for different wnidows;
              // they need a distinct id. If you include multiple items,
              // the scene property must be decorated with "@SceneBuilder" as above.
              WindowGroup(id: "SimpleText") {
                  Text("Hello World")
              }
          }
      }

HelloWorldContentView.swift`

`

          struct PHPickerViewControllerWrapper: UIViewControllerRepresentable {
              @Binding var image: UIImage?
              @Binding var videoURL: URL? // 添加视频URL绑定
              let isSelectingImage: Bool // 是否选择图片
              @Environment(\.presentationMode) var presentationMode
              
          func makeCoordinator() -> Coordinator {
              return Coordinator(parent: self)
          }
          
          func makeUIViewController(context: Context) -> PHPickerViewController {
              var configuration = PHPickerConfiguration()
              if isSelectingImage {
                  configuration.filter = .images
              } else {
                  configuration.filter = .videos
              }
              configuration.selectionLimit = 1
              let picker = PHPickerViewController(configuration: configuration)
              picker.delegate = context.coordinator
              return picker
          }
          
          func updateUIViewController(_ uiViewController: PHPickerViewController, context: Context) {
              // Nothing to update
          }
          
          class Coordinator: NSObject, PHPickerViewControllerDelegate {
              let parent: PHPickerViewControllerWrapper
           
              init(parent: PHPickerViewControllerWrapper) {
                  self.parent = parent
             
              }
              
              func picker(_ picker: PHPickerViewController, didFinishPicking results: [PHPickerResult]) {
                      parent.presentationMode.wrappedValue.dismiss()
                      
                      for result in results {
                          let itemProvider = result.itemProvider
                          // 检查是否可以加载图片或视频
                          if parent.isSelectingImage && itemProvider.canLoadObject(ofClass: UIImage.self) {
                              // 加载图片
                              itemProvider.loadObject(ofClass: UIImage.self) { [weak self] image, error in
                                  if let image = image as? UIImage {
                                      DispatchQueue.main.async {
                                          self?.parent.image = image
                                          if let data = image.jpegData(compressionQuality: 1.0) {
                                              let filename = UUID().uuidString + ".jpg"
                                              let documentsDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
                                              let fileURL = documentsDirectory.appendingPathComponent(filename)
                                              do {
                                                  try data.write(to: fileURL)
                                                  // 调用 Objective-C 中的方法将图片路径传回 Unity
                                                  CallCSharpCallback(fileURL.path,index: 1)
                                              } catch {
                                                  print("Error writing image data to disk: \(error)")
                                              }
                                      }
                                      }
                                  }
                              }
                          } else if !parent.isSelectingImage && itemProvider.canLoadObject(ofClass: URL.self) {
                              // 加载视频
                              itemProvider.loadObject(ofClass: URL.self) { [weak self] videoURL, error in
                                  if let videoURL = videoURL as? URL {
                                      DispatchQueue.main.async {
                                          self?.parent.videoURL = videoURL
                                      }
                                  }
                              }
                          }
                      }
                  }
          }
      }

`

[yasirkula]

@yosun Thanks again 🌷 This solution probably won't apply to most Unity users but hey, it works for you! I couldn't see AI models in action in the video though I'm sure we'll be seeing them soon.

@414726193 That's a comprehensive Swift answer, thank you for sharing your findings!