Transferable?

[dsanders11]

Would be useful if the ImageCapture interface was [Transferable], as this would allow offloading grabbing frames from the main thread. Trying to grab frames on the main thread is always going to compete with UI events and rendering code, which can interfere with the tight timings needed to try to maintain 30 FPS frame grabbing.

[alvestrand]

ImageCapture gives you an encoded image (probably a JPEG). It's optimized for still image capture.
What are you trying to accomplish, and can it be described as an use case in webrtc-nv-use-cases?

[dsanders11]

ImageCapture gives you an encoded image (probably a JPEG). It's optimized for still image capture.

@alvestrand, ImageCapture.grabFrame is, AFAIK, not optimized for still image capture.

What are you trying to accomplish, and can it be described as an use case in webrtc-nv-use-cases?

I've had a use-case for many years of doing various detections from webcam video data (barcode detection, a lightweight neural network, etc). They require getting an ImageData for a webcam video frame, which ImageCapture now provides an efficient way of doing via ImageCapture.grabFrame. I also need the ability to control the camera (focus, zoom, pan and tilt), which ImageCapture also provides.

Looking at your link, the closest would probably be:

   N18             The application must be able to obtain raw media
                   from the capture device in desired formats.

However that may not be entirely true - I don't want true raw media, as then I'd have to deal with the various formats a webcam stream can be. At the end of the day I want an ImageData since it's a consistent format, and can easily be provided to any library expecting image data.

[riju]

@dsanders11 : I have used OpenCV.JS to accomplish similar use cases, like card scanning. Please have a look at our WebCamera project.
We did try out getting canvasContext.getImageData to a SharedArrayBuffer and used worker.postMessage and did the computations in the workers. It's not yet on master as the WASM (with SIMD + workers) optimizations improved the user perceived performance to an acceptable level.

@alvestrand : The webrtc-nv-use-cases look good and I am interested mainly in the requirement -

   N18             The application must be able to obtain raw media
                   from the capture device in desired formats.

and what developers can do with that.
Like @dsanders11 said, it can be sub-optimal to deal with various formats a webcam can stream, maybe the WebCodecs proposal can be of help ?

[riju]

Does this sound similar to how audio processing is done in WebAudio ? Should we be then thinking about Worklets then ?

I think there was something similar happening at https://w3c.github.io/mediacapture-worker/ but was discontinued.