Many problems with video presentation

[rkmaddox]

So my labbies and I are having a heck of a time getting video to work. Here is a summary of the issues as I understand them, though they will fill in the rest.

0. AVBin will not play quicktime movies (we can deal with that, will just switch to x264)
1. AVBin freezes near (but not quite at) then end of our video (x264 encoded)
2. AVBin has strange image quality (Sara will post screen cap). To me it looks like a missed keyframe, but it never fixes itself. When the video is played in a media player it looks fine.

Those are the problems with trying to play a movie in pyglet. Strangely, the issues don't appear in the simple_video example.

Another, bigger, issue, which @drammock alluded to in #273 is that the sync may be off when playing videos. What we really need is access to the video frames and the ability to get the textures and flip as needed. This would also solve the problem of getting it in sync with the audio and triggers. Basically, we would handle video as we did in 1v2a, but instead of saucers it would be video frames. We have tried very hard to pull textures from videos in pyglet, but it is not working for us yet.

Labbies are currently signing up for github and will post more info here.

[rkmaddox]

This will also be essential to figure out in order to present video on the monitor with the corresponding audio from the TDT.

[drammock]

For the record, I never got it satisfactorily working in my initial attempts, so I'm not surprised you're struggling. I bailed when I realized I didn't actually need AV synchrony, and used a hacky solution of hiding the expyfun window and popping up a borderless full-screen VLC window to play the video. That doesn't mean we can't get this to work, but it will probably require some additions to / overhaul of the video code.

[rkmaddox]

For the time sync, would providing our own clock with t=0 as the start time help?

[rkmaddox]

@drammock Yeah, it is pretty much essential for us now. We have many upcoming experiments where playing natural videos is the only way to go, and they have to be time synced.

[rkmaddox]

The sync itself should not be difficult. It was essentially solved with 1v2a. What we need to be able to do is just use a pyglet video source as a container for a sequence of frames, pull the frames out one by one, and flip them at the time we deem correct.

[rkmaddox]

Sooooooo. I just took maybe 45 seconds to see if I could do what I want to do with opencv. I can. I already got it working.

I don't love the idea of adding a dependency, but damn. No AVBin, no trouble, just reading frames in. And it happens way way faster than real time, so it shouldn't hold anything up.

[sfiscell]

This is an image of the video quality:

[rkmaddox]

Here is a very siomple example that reads in all the frames of a video using opencv:

vid = cv2.VideoCapture('/path/to/video.mp4')
frames = []
success = True
while success:
    success, frame = vid.read()
    if success:
        frame = frame[:, :, ::-1]
        frames += [frame]

Note that your RAM will die if you do a long video. There is no need in practice to store all the frames.

[larsoner]

When you figure out how to do this with opencv, can you turn it into an example? Bonus points if you can find and use the original McGurk stimuli to do it with.

[rkmaddox]

Are you saying we should use opencv to do it? We would still use pyglet to show the frames, but would load them using opencv. In other words: do we have the go ahead to add opencv as a dependency (at least if you want to do video playing)?

[larsoner]

Yeah that seems like a reasonable approach to me

[drammock]

@sfiscell you probably already know this, but in case not: the video quality probably arises because (1) the chosen video codec only encodes pixels that changed from the previous frame, and (2) the expyfun flip operation clears all those non-changed pixels before displaying the new frame. Results might be better with a video format like AVI, which encodes every pixel in every frame. But if we want to get this working quickly and support a wider range of formats, OpenCV may be the way to go.

[rkmaddox]

Here is a gist that gives the idea of what I am thinking of.

It should not be considered an example of how to do this.

I need to figure out how to get the timing right and always stay one frame ahead and skip frames when necessary etc etc. But this should at least make Mark's head spin.

[rkmaddox]

I plan to work more on this tomorrow to make it reasonable.

[rkmaddox]

Update on this: OpenCV works great for loading the frames into numpy arrays.

however, getting these into pyglet RawImage objects (even if we try to write over the texture directly) takes much longer than a frame, so we cannot do it in real time. This leads to about 3 or 4 s worth of video loading before every trial, which really isn't workable.

Our current thought is perhaps pickling the textures right before running the experiment to avoid converting them during. But would that work, or does making the texture also load stuff into GPU memory? I'm frankly shocked that starting with the data in the proper format it could take so long just to get it on the screen.

Any recommendations greatly appreciated--we are slated to start collecting data early next week, and @sfiscell needs to have a respectable number of subjects collected by the end of the week to present for her summer project's final poster.

[larsoner]

It needs to upload to the GPU, so no pickle. The slow part could be the shader compilation not data upload. Did you try creating a single RawImage and just updating the new data? That should be fast enough I hope

…

[larsoner]

... although looking at the code, even set_image instantiates a new Sprite, which is probably not good.

What probably would be fast is using ImageData because it has a set_data method:

http://pyglet.readthedocs.io/en/pyglet-1.3-maintenance/modules/image/index.html

[rkmaddox]

Have tried something along those lines, didn't seem to help. You know what's funny though? Despite being in the docs, there is no set_data method in the AbstractImage class in pyglet 1.3.2.

[rkmaddox]

Hold on, I may be wrong on that. Will be back.

[rkmaddox]

It seems like the .to_string() call is one of the very slow parts, yet it seems to be required by pyglet. The image data is already in uint8 so it seems like changing that to string should be extraordinarily fast.

[rkmaddox]

To elaborate, you are right that using set_data is very fast, but now the tricky part is getting it into a texture so that it can be drawn.

[rkmaddox]

Putting this here in case it helps: https://github.com/motmot/pygarrayimage. Reference in this stack overflow post: https://stackoverflow.com/questions/31030000/capture-webcam-image-using-cv2-and-pyglet-in-python.

[larsoner]

FYI @rkmaddox Pyglet 1.4 is out. It no longer uses AVBin but rather FFmpeg. Maybe it works better.

But if we want to use that there is a lot more work to do because they have some backward incompatible stuff (#385).

I was planning to / hoping to get rid of Pyglet. But if you find that the FFmpeg video replay actually works, then that would be a good reason to support it.

[drammock]

the avbin / ffmpeg change is a good thing. It turns out there was a brief schism in the ffmpeg dev community, avbin was a splinter project that was (briefly) better, but has since been abandoned / merged back into ffmpeg. Whether it's good enough when used through pyglet to do what @rkmaddox wants is a separate question, but if you want to test it I think we could (should?) switch our dependency back from avbin to ffmpeg without updating the pyglet dependency (unless pyglet 1.3 hardcodes avbin? I forget).

[larsoner]

I don't think there is an FFmpeg source in Pyglet < 1.4

https://bitbucket.org/pyglet/pyglet/src/pyglet-1.3.2/pyglet/media/sources/

[rkmaddox]

OK. I probably won't think about this until next time we need video. This will only help with loading videos, though, not presenting them right? Either way that will be good, since I think we ended up converting all our videos to images and then loading those in before each trial, which did introduce a longer inter-trial pause than was optimal.

[larsoner]

FYI Pyglet has revamped their support to use FFMpeg and I recently fixed a bug with video in #413. I'm going to assume this will magically fix everything and mark that PR to close this one, but let's reopen if there continue to be issues. @rkmaddox feel free to try (or have others try) with the new code!

[larsoner]

FYI @rkmaddox video should work much better on all platforms on latest expyfun once a new 1.5.22 pyglet is cut, or you use pyglet-1.5-maintenance branch like pip install https://github.com/pyglet/pyglet/zipball/pyglet-1.5-maintenance