Face Detection MediaStreamTrack constraints and VideoFrame metadata

[ttoivone]

Request for Mozilla Position on an Emerging Web Specification

• Specification Title: Face Detection
• Specification or proposal URL: https://github.com/w3c/mediacapture-extensions/blob/main/face-detection-explainer.md
• Caniuse.com URL (optional):
• Bugzilla URL (optional):
• Mozillians who can provide input (optional): Jan-Ivar Bruaroey

Other information

Updated PR based on feedback and the new WebCodecs VideoFrameMetadata dictionary is here:
mediacapture-extensions #78.

[zcorpan]

cc @jan-ivar

[jan-ivar]

Sorry for the late response. I don't have a strong opinion here.

This is not facial recognition, rather it detects where faces are (eyes, mouth). Its purpose would be optimizing quality and power consumption of video effects like background replacement and other augmentation popular in video conferencing.

This seems somewhat useful in providing parity with non-web applications in providing hardware assistance (from drivers/OS etc) with detecting outlines of heads in camera video, where available.

Privacy-wise the segment information does not appear to reveal anything about a videoframe that couldn't be deduced through processing by the application itself. It might reveal specifics about a particular camera driver or OS, but this seems minor compared to other information exposed by an active camera and its device label.

It would only benefit applications that are aware of and utilize this API. I don't have any info on how prevalent hardware is, nor existing usage in other browsers. Absent that, a criticism might be why this functionality is tied to a camera source and that time might be better spent on a broader API that can process web-provided frames as well, like the one mentioned in the explainer. But if that's where the technology exists, that's where it is.