Number of frames interpreted by Lintel is not correct

[HaoMood]

Dear Brendan,

Thanks for this wonderful project for video loading. It saves great amount of space and time consumption.

However, I found that Lintel may not parse the number of frames of a video correctly. I install Lintel by conda install:
conda install -c conda-forge lintel

Case 1: AVI data.

import cv2, lintel
path = 'v_ApplyEyeMakeup_g01_c01.avi'
video = cv2.VideoCapture(path)
num_frames = int(video.get(cv2.CAP_PROP_FRAME_COUNT)) 
video.release()
result = lintel.loadvid_frame_nums(open(path, 'rb').read(), frame_nums=(64,))

The number of frames is 164, while the above line gives "Ran out of frames. Looping. No frames received after seek." error.

Case 2: MP4 data.

import cv2, lintel
path = 'x0xr91NbD9E.mp4'
video = cv2.VideoCapture(path)
num_frames = int(video.get(cv2.CAP_PROP_FRAME_COUNT)) 
video.release()
result = lintel.loadvid_frame_nums(open(path, 'rb').read(), frame_nums=(808,))

The number of frames is 809, while the above line gives "Ran out of frames. Looping. No frames received after seek." error.

The frame numbers returned by OpenCV is correct, which I have double checked by using the following command:
fmpeg -i v_ApplyEyeMakeup_g01_c01.avi -vcodec copy -f rawvideo -y /dev/null 2>&1 | tr ^M '\n' | awk '/^frame=/ {print $2}'|tail -n 1

In short, the number of frames interpreted by Lintel is nearly correct for MP4 video, while it has a huge disparity for AVI video. I can provide these two example videos, if necessary.

Thank you for your kind consideration.

[dukebw]

Hi Hao,

Thank you for taking the time to test this out. I was aware that the MP4 frame counting was off by 1 frame, but I didn't know about the AVI container being completely wrong. Would you mind hosting the AVI file, if it's no trouble? I will take a look as soon as possible.

[HaoMood]

Thanks for your immediate reply.

I will send the AVI video to your Gmail account.

By the way, for MP4 video data, only a few frames are missed by Lintel. Is there any way to know in advance how many frames can Lintel read from? That is to say, if I do not want to encounter a run-time error for "Ran out of frames. Looping. No frames received after seek.", what can I do? One possible way is

video, width, height, seek_index = lintel.loadvid(open(file, 'rb').read(), should_random_seek=False)
video = np.reshape(np.frombuffer(video, dtype=np.uint8), (-1, height, width, 3))
num_frames = video.shape[0]

However, it needs to load the entire video. It is slow to get the frame number. Is there a better solution?

Thanks for your kind consideration.

[dukebw]

I think for this case, what you could do is preprocess all the files:

1. Get the number of frames from ffprobe.
2. Work backwards from what should be the last frame, until Lintel stops printing the "Looping" error message.
3. Record this as metadata for each video.

You could probably script this in Python by piping stderr to a file and parsing it, or something like that.

Sorry again about the frame counting issue. I will start looking at it tonight and see if I can quickly fix it.

[HaoMood]

Thanks for your immediate reply.

[dukebw]

Hi, I tried your .avi file and you are right, Lintel doesn't read frames correctly using the AVI container.

However, if I convert to MP4 using this command:

ffmpeg -i filename.avi filename.mp4

Lintel seems to count frames correctly both using seek and not using seek.

I counted frames with this FFmpeg command:

ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_frames -of default=nokey=1:noprint_wrappers=1 ./filename.mp4

This gave me 165. When I tried loading the 165th frame (frame index 164) using:
lintel_test --filename ./filename.mp4 --dynamic-size --frame-nums --should-seek --start-frame 164

I don't get the "No frames received after seek" message, meaning that Lintel is able to decode the last frame. Note that I am using the latest master branch from the repository.

If you send me your MP4 file (the one with 809 frames) I can try to reproduce the issue on that one, but I tried about a dozen videos on my machine that all seem to work.

[HaoMood]

Thanks for your kind reply.

Converting AVI to MP4 in advance works.

I think the lintel on Anaconda cloud is some kind of out-dated. Lintel installed by conda still reports "No frames received after seek" error for the 165th frame (frame index 164) on MP4. So, it will be appreciate if you consider updating the conda as well.

Thanks.

[AlexHu123]

hello haomood, i still face the same problem as yours, i use video from HMDB-51 dataset (originally .avi, i use ffmpeg filename.avi filename.mp4 to convert to mp4 video), but i still find ran out of frames, looping.

[HaoMood]

Actually, Lintel will miss the first frame of a video. Considering the following code, named test_lintel.py, it will generate a dummy video, which has five frames. Each frame is a pure colored image, which are black, white, red, green, and blue, respectively.

# -*- coding: utf-8 -*-
"""Generate fake video and test Lintel."""


import os
from typing import List

import cv2
import lintel
import numpy as np


def generateDummyVideoFrames(video_path: str):
    """Generate a video for testing."""
    video_frames = []

    # 0th frame: Black.
    frame = np.zeros((224, 224, 3)).astype(np.uint8)
    video_frames.append(frame)

    # 1st frame: White.
    frame = np.ones((224, 224, 3)).astype(np.uint8) * 255
    video_frames.append(frame)

    # 2nd frame: Red.
    frame = np.zeros((224, 224, 3)).astype(np.uint8)
    frame[:, :, 0] = 255
    video_frames.append(frame)

    # 3rd frame: Green.
    frame = np.zeros((224, 224, 3)).astype(np.uint8)
    frame[:, :, 1] = 255
    video_frames.append(frame)

    # 4th frame: Blue.
    frame = np.zeros((224, 224, 3)).astype(np.uint8)
    frame[:, :, 2] = 255
    video_frames.append(frame)

    # Write video.
    f = cv2.VideoWriter(video_path, cv2.VideoWriter_fourcc(*'mp4v'), 1, (224, 224))
    for frame in video_frames:
        frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
        f.write(frame)


def lintelReadFramesFromIndex(video_path: str, index: List[int]) -> np.ndarray:
    """Decord video using Lintel."""
    assert os.path.isfile(video_path)
    with open(video_path, 'rb') as f:
        video, width, height = lintel.loadvid_frame_nums(
            f.read(), frame_nums=index)
    video = np.reshape(np.frombuffer(video, dtype=np.uint8),
                       (len(index), height, width, 3))
    return video


def main():
    """The main function."""
    video_path = './debug.mp4'
    generateDummyVideoFrames(video_path)
    video = lintelReadFramesFromIndex(video_path, [0])
    print(video)


if __name__ == '__main__':
    main()

Run this code using python test_lintel.py, it outputs

[[[[251 253 250]
   [251 253 250]
   [251 253 250]
   ...
   [251 253 250]
   [251 253 250]
   [251 253 250]]

  [[251 253 250]
   [251 253 250]
   [251 253 250]
   ...
   [251 253 250]
   [251 253 250]
   [251 253 250]]

  [[251 253 250]
   [251 253 250]
   [251 253 250]
   ...
   [251 253 250]
   [251 253 250]
   [251 253 250]]

  ...

  [[251 253 250]
   [251 253 250]
   [251 253 250]
   ...
   [251 253 250]
   [251 253 250]
   [251 253 250]]

  [[251 253 250]
   [251 253 250]
   [251 253 250]
   ...
   [251 253 250]
   [251 253 250]
   [251 253 250]]

  [[251 253 250]
   [251 253 250]
   [251 253 250]
   ...
   [251 253 250]
   [251 253 250]
   [251 253 250]]]]

which is the white frame, not the black frame.

Change

video = lintelReadFramesFromIndex(video_path, [0])

to

video = lintelReadFramesFromIndex(video_path, [1])

gives the red frame, [2] gives the green frame, [3] gives the blue frame, and [4] gives the

Ran out of frames. Looping.
No frames received after seek.

error.

Acknowledgement of Jian Zhou ([email protected]) for revealing this issue.