(GL|MT)HKView rendering is delayed with .preferredVideoStabilizationMode: AVCaptureVideoStabilizationMode.auto

[animaonline]

Describe the bug
The GLHKView seems to fall behind and does not render in realtime when using cinematic/auto video stabilization, it seems that the rendering is ~300-500ms delayed

Related to: #574

To Reproduce
Easily reproduced in the example app

Expected behavior
The GLHKView renders without a delay, just like the HKView

Smartphone (please complete the following information):

• Device: iPhone X
• OS: iOS 12.4.1

[CmST0us]

It seems HKView(which use AVCaptureVideoPreviewLayer to render SampleBuffer) connects to AVCaptureSession directly(something like camera's viewfinder), and it's difference from rending SampleBuffer that comes from
captureOutput:didOutputSampleBuffer:fromConnection of AVCaptureVideoDataOutput.
This looks like some Apple feature - _ -

Or you can try to create an AVCaptureVideoPreviewLayer, and set it's session property. As the code below.

@property(nonatomic, strong) AVCaptureVideoPreviewLayer *enhanceCamera;
@property(nonatomic, strong) AVCaptureSession *captureSession;

- (void)setupCamera {
    self.enhanceCamera = [[AVCaptureVideoPreviewLayer alloc] init];
    self.enhanceCamera.session = self.captureSession;
    /// Yes, just this two lines, you even no need to add the AVCaptureVideoPreviewLayer to layer...
}

In my project, it work! but I don't know why - . -

[mkrn]

I've noticed that when setting .preferredVideoStabilizationMode: AVCaptureVideoStabilizationMode.auto
the streamed audio runs 300ms-1s ahead of the video .. This is more of an issue, so had to turn it off.
Any ideas what might be causing it?

[animaonline]

@mkrn I'm having the same exact problem, did you find any solutions to this?

[mkrn]

@animaonline I've set .preferredVideoStabilizationMode: AVCaptureVideoStabilizationMode.standard.rawValue
It introduces no delay.
According to documentation the .cinematic and .cinematicExtended add considerably more latency to the video capture pipeline.
Setting to .auto is fine on some devices but delayed on other devices.

Not sure if that can be offset somehow so audio has some latency to match video / audio, @shogo4405 ?

@CmST0us trick didn't help me, unfortunately (adding this to init:)

    self.enhanceCamera = AVCaptureVideoPreviewLayer();
    self.captureSession = AVCaptureSession();
    self.enhanceCamera.session = self.captureSession;

[CmST0us]

@animaonline I've set .preferredVideoStabilizationMode: AVCaptureVideoStabilizationMode.standard.rawValue It introduces no delay. According to documentation the .cinematic and .cinematicExtended add considerably more latency to the video capture pipeline. Setting to .auto is fine on some devices but delayed on other devices.

Not sure if that can be offset somehow so audio has some latency to match video / audio, @shogo4405 ?

@CmST0us trick didn't help me, unfortunately (adding this to init:)

    self.enhanceCamera = AVCaptureVideoPreviewLayer();
    self.captureSession = AVCaptureSession();
    self.enhanceCamera.session = self.captureSession;

I find this trick did not work in newly iOS......It looks like Apple has fixed the problem.(Good job Apple..)

AVCaptureConnection is something like ISP in a camera system, which connect the sensor and encoder(or cpu?). When we set the .preferredVideoStabilizationMode, actually, it will set some property in ISP.
So it will cost more time to process a stabilization image.

[mkrn]

@shogo4405 any ideas to sync Audio and Video capture with stabilization using AVCaptureVideoStabilizationMode.cinematic or AVCaptureVideoStabilizationMode.cinematicExtended ?

It would be really nice to have

The sync issue is present only in the Live stream. The audio/video is synced in the local recording perfectly.

[mkrn]

@shogo4405 In the recorded video, the audio and video are synced. In the live stream, the audio goes Ahead of the video. Hope it helps

[shogo4405]

I fixed it this issue (out of sync). #1012 (comment)

[shogo4405]

If I set this property.
.sessionPreset = .vga640x480
.sessionPreset = .cif352x288
.sessionPreset = .low

It seems that without latency. But poor video quality.
I have no idea decrease latency.

I can sync a video and audio in the livestream.
I will implement it.

[shogo4405]

It works fine audio and video synchronizatio. #1000 Need more testing...