Random FPS drop issue with OpenCVSharp using Cv2.Resize (May not related).

[famasf1]

Summary of your issue

My app is Winforms Desktop app using OpenCV to record and stored video files. I'm currently using

VideoCapture videoCapture = VideoCapture.FromCamera(index: cameraIndex, apiPreference: VideoCaptureAPIs.DSHOW);

And it worked fine for the most part. But then about few days ago, My client reported that sometimes after the app loaded and the camera is initialized, The FPS dropped to basically 1-5FPS and look very choppy. I'm able to reproduced by repeatedly re-initialized VideoCapture and out of nowhere it just happened. There is a hack-y workaround and that is close the app and reopen it or repeatly re-initialized the VideoCapture. But i would like a more permanent solution. So i've been basically spending 2 full days trial & error my way to hunting them down. But no luck.

Although I found out that if i change my apiPreferences to VideoCaptureAPIs.MSMF the issue is basically solved. Plus the performance gain is very noticable. But it leads to another problem similar to issue #1663. So now i'm really ran out of options. I also not entirely sure that the API is the one causing the issue, and it might be on my part. But i'm basically hitting the real dead-end here.

I'll try to reproduced with an example. But in the meantime if you can, take a look at my snippet. You may know something that i don't.

Environment

Windows 11 64-Bit.

What did you do when you faced the problem?

Literally opening up the app and start the camera initialization. It happened randomly everytime the camera need to be initialize. Maybe it's memory issue?

Example code:

Create VideoCapture in OpenCVHelper.cs

public VideoCapture CreateVideoCaptureFromCamera(int cameraIndex, int width, int height, double inputFPS, int IsAutoFocus)
{
    try
    {
        VideoCapture videoCapture = VideoCapture.FromCamera(index: cameraIndex, apiPreference: VideoCaptureAPIs.DSHOW);
        videoCapture.FrameHeight = height;
        videoCapture.FrameWidth = width;
        videoCapture.Fps = inputFPS;

        if (IsAutoFocus == (int)AutoFocusStatus.On)
        {
            videoCapture.AutoFocus = true;
        }
        else
        {
            videoCapture.AutoFocus = false;
            videoCapture.Focus = 100;
        }
        videoCapture.Zoom = 0;
        videoCapture.FourCC = "MJPG";
        return videoCapture;
    }
    catch
    {
        throw;
    }
}

Now in Presenter layer.

videoCapturePresenter = recorderMainAppManager.CreateVideoCapture(
    index: camera.Index,
    resolution: resolution, //resolutions[resolutions.Count - 1],
    fps: ProfilePresenter.CameraFps,
    focus: ProfilePresenter.IsAutoFocus ?? (int)AutoFocusStatus.Off
);
captureTokenSource = new();
_captureThread = new Thread(() => CaptureFrame(videoCapturePresenter, captureTokenSource))
{
    Name = "Capture Thread",
    Priority = ThreadPriority.AboveNormal, ///This is not the cause because i added after i can reproduced the issues. 
};
_captureThread.Start();

This is CaptureFrame. I used Cv2.Resize to resize the frame before bringing it to the UI layer. Without it PictureBox will attempted to resize each frame by itself and take half of my available RAM along with it.

private void CaptureFrame(VideoCapture capture, CancellationTokenSource captureToken)
{
    
    string newOverlayText = string.Empty;
    try
    {
        ShipmentOverlay shipmentOverlay = recorderMainAppManager.GetShipmentOverlay
            (
                resolution: $"{capture.FrameWidth}x{capture.FrameHeight}"
            );

        string username = $"Employee Name : {UserPresenter.Username}";
        while (!captureToken.IsCancellationRequested)
        {
            string datetime = DateTimeOffset.Now.ToString("dd/MM/yyyy HH:mm:ss");
            string overlayText = createNewOverlayString();
            using (Mat rawFrame = new())
            {
                newOverlayText = $"Date : {datetime} | {username} | {overlayText}";
                if (!capture.Read(rawFrame))
                {
                    Console.WriteLine("Error: Failed to read frame.");
                    break;
                }

                if (rawFrame.Empty())
                {
                    Console.WriteLine("Error: Empty frame.");
                    break;
                }

                using (Mat frameWithOverlay = recorderMainAppManager.DrawOverlay(frame: rawFrame, shipmentOverlay: shipmentOverlay, overlayText: newOverlayText))
                {
                    Mat clonedFrame = frameWithOverlay.Clone();
                    {
                        lock (clonedFrame)
                        {
                            var currentPictureBox = recorderMainAppView.GetPicRecordResolution();
                            var targetSize = new OpenCvSharp.Size() { Width = currentPictureBox.Width, Height = currentPictureBox.Height };
                            if (targetSize.Width > 0 && targetSize.Height > 0)
                            {
                                Mat captureMat = new();
                                Cv2.Resize(clonedFrame, captureMat, targetSize, interpolation: InterpolationFlags.Area);
                                using (Bitmap _bitmap = BitmapConverter.ToBitmap(captureMat))
                                {
                                    Bitmap clonedResizeFrame = (Bitmap)_bitmap.Clone();
                                    recorderMainAppView.UpdateCameraView(clonedResizeFrame);
                                }

                                if (_isRecording == RecordStatus.RECORD)
                                {
                                    _frameQueue.Enqueue(clonedFrame);
                                }
                                else
                                {
                                    captureMat.Dispose();
                                }
                            }
                        }
                    }
                }
            }
            _threadStopEvent.Wait(1);
            if (captureToken.IsCancellationRequested) break;
        }
    }
    catch
    {
        throw;
    }
}

This is UI layer. When my camera start producing a frame, it will be sent to PictureBox as video preview for my user.

public void UpdateCameraView(Bitmap img)
{
    try
    {
        if (picRecord.InvokeRequired && 
            WindowState != FormWindowState.Minimized)
        {
            using (picRecord.Image)
            {
                Invoke(() => picRecord.Image = img);
            }
        }
    }
    catch (ObjectDisposedException)
    {
        // This function throw everytime in debug mode, apparently because some video frame aren't being disposed yet
        // But frmMain is already disposed. Happened only when i exit the app.
        // For now, this is working. But if you know how to solve this. Feel free.
        return;
    }
}

Output:

video output that sometimes run fine. Sometimes FPS drop to 1. The log always output `30` however.

What did you intend to be?

Video start without any hitches or slowdown.