Desktop Capture Input and example #79

vicnevicne · 2020-06-08T23:30:11Z

Follow-up to issue #78

…Apache commons' logic)

kokorin

It's also required to add ScreenCaptureInput tests.
This project uses Github Actions for testing, but I'm not sure if it's possible to have Desktop VM (i.e. non headless) there.

Probably https://github.com/marketplace/actions/screenshots-ci-action is good start point.

src/main/java/com/github/kokorin/jaffree/ffmpeg/DesktopCaptureInput.java

src/main/java/com/github/kokorin/jaffree/OS.java

kokorin · 2020-06-10T07:07:06Z

src/test/java/examples/ffmpeg/DesktopCapture.java

@@ -0,0 +1,59 @@
+package examples.ffmpeg;


Add please Apache v2 licence.

Done.
Please note most other examples don't have a license boilerplate though...

Yes, I have to add them, my bad

The same as to previous Apache Licence - put your credits and 2020 year

vicnevicne · 2020-06-10T19:43:44Z

Hi, and thanks for the review.
As you know, my first goal is to discover and learn Jaffree (and FFmpeg along the way), so this code is not ready for prime-time I guess. I would like your advice on a few things like:

the right way to include options with platform-dependent support. For example producing a stream for only a given area is supported on Windows with GDIgrab and Linux, but seemingly not for Windows with DirectShow and Mac, in which case cropping must be done by an additional filter. What do you think the API should look like? Should we throw an exception or just ignore the call when the option is not supported?
audio is supported in different ways, but audio device selection must be made in a list of available devices. This list can be produced by a specific ffmpeg call (e.g. ffmpeg -list_devices true -f dshow -i dummy). What would be the most elegant way to retrieve that list using Jaffree ?
etc.

Anyway, I will fix the license and OS right away. It might take longer to setup a test but I'll give it a go.

vicnevicne · 2020-06-10T20:49:38Z

I added a test for DesktopCaptureInput and it succeeds on my Windows PC:

c:\utils\ffmpeg\bin\ffmpeg -f gdigrab -r 10 -video_size 160x120 -offset_x 80 -offset_y 80 -i desktop -y -t 10.000 C:\Users\Vincent\AppData\Local\Temp\jaffree43154375680172906\desktop.mp4
INFO main com.github.kokorin.jaffree.process.ProcessHandler - Starting process: c:\utils\ffmpeg\bin\ffmpeg
INFO main com.github.kokorin.jaffree.process.ProcessHandler - Waiting for process to finish
INFO main com.github.kokorin.jaffree.process.ProcessHandler - Process has finished with status: 0

I doubt it will succeed in the automated Github environment though, due to the headless issue you mention. Unfortunately, the page you linked seems to be only about grabbing browser output (I think what they call "desktop" is just "desktop browser" as opposed to "mobile browser").

Maybe https://github.com/marketplace/actions/gabrielbb-xvfb-action would be an option, but I have no idea how to integrate that with the test framework you have put in place.

Alternately, we can leave the test in @ignore with an explanation, like for the testAlpha() method...

Anyway, as mentioned above, the current implementation runs on Windows (GDI mode) and most probably on Linux, but will not yet crop the requested area on Mac and Windows (DirectShow mode), so a successful test on Linux is not an indication of overall correctness.

Edit: Confirmed: automated tests fail with
java.lang.RuntimeException: ffmpeg exited with message: :0.0+80,60: Input/output error
I'm marking that test with an @ignore annotation for now...

kokorin · 2020-06-11T07:10:01Z

the right way to include options with platform-dependent support. For example producing a stream for only a given area is supported on Windows with GDIgrab and Linux, but seemingly not for Windows with DirectShow and Mac, in which case cropping must be done by an additional filter. What do you think the API should look like? Should we throw an exception or just ignore the call when the option is not supported?

That case with adding extra arguments as ffmpeg filter seems to be too complex for Jaffree. Honestly I don't want such changes to be added to ScreenCaptureInput. Such complex behaviour is more suiteable for custom code or even separate library which will provide extra capabilities on top of the Jaffree.

audio is supported in different ways, but audio device selection must be made in a list of available devices. This list can be produced by a specific ffmpeg call (e.g. ffmpeg -list_devices true -f dshow -i dummy). What would be the most elegant way to retrieve that list using Jaffree ?

The same as above. But it could be useful to add Input implementation which is capable of getting audio/video device list.

vicnevicne · 2020-06-11T12:22:34Z

I agree with you, better keep Jaffree simple and move complex configurations to an extra layer.
In case an option (e.g Area selection) is not supported on a platform, I was wondering if the request should be ignored or throw an exception, but I think the best solution would be to ignore it but log a message stating it is being ignored and suggesting an alternative. I'll do that change.

Now for audio, the issue is not that much for the Input implementation to select the right device, but more about an API (probably not an Input) that would query FFmpeg and return the list of devices, the same way FFprobe returns the list of Streams in a file.
Does Jaffree already have a similar "query" usage that I could mimick ?
Note that it's not just for audio btw. Screen capture is also just one device among others (e. g. webcams).

Now again, support is very dependant on the platform. DirectShow (Windows) and avfoundation (Mac) support device enumeration but I don't think X11Grab (Linux) or GDIGrab (Windows alt) do. Sounds logical as they are targeted at the display... What do you think ?

More generally speaking, does Jaffree expose a "low level" API that would allow a client app to build its own FFmpeg "recipe" while benefiting from the wrapping layer of Jaffree ?

src/main/java/com/github/kokorin/jaffree/ffmpeg/DesktopCaptureInput.java

kokorin · 2020-06-11T07:12:39Z

src/main/java/com/github/kokorin/jaffree/ffmpeg/DesktopCaptureInput.java

+ * - Call ffmpeg to return list of devices? list of screens?
+ */
+public class DesktopCaptureInput extends BaseInput<DesktopCaptureInput> implements Input {
+    private final boolean WINDOWS_USE_GDI = true;


Better to introduce an enum with DIRECT_SHOW and GDI values. Also since it's not static valirable it's name should be in snakeCase.

Oops, I meant it to be a static configuration.
In fact, I think it should disappear in the long run but I'm still not convinced which one is best. GDI allows cropping but DirectShow seems the way to go for audio.
If we could mix GDI for video and Directshow for audio, I think it would be perfect, but I have to test further. That would prevent other sources such as webcams however, but we're in a Desktop Capture context anyway.
If the conclusion is that we need to keep both modes, I'll add a setter and use an enum indeed.

kokorin · 2020-06-11T07:15:40Z

src/test/java/examples/ffmpeg/DesktopCapture.java

@@ -0,0 +1,59 @@
+package examples.ffmpeg;


The same as to previous Apache Licence - put your credits and 2020 year

kokorin · 2020-06-11T13:02:47Z

Now for audio, the issue is not that much for the Input implementation to select the right device, but more about an API (probably not an Input) that would query FFmpeg and return the list of devices, the same way FFprobe returns the list of Streams in a file.
Does Jaffree already have a similar "query" usage that I could mimick ?
Note that it's not just for audio btw. Screen capture is also just one device among others (e. g. webcams).

No, there is no such FFprobe-like output parsing, but there is an option to parse ffmpeg non-progress output. Check com.github.kokorin.jaffree.ffmpeg.OutputListener

Now again, support is very dependant on the platform. DirectShow (Windows) and avfoundation (Mac) support device enumeration but I don't think X11Grab (Linux) or GDIGrab (Windows alt) do. Sounds logical as they are targeted at the display... What do you think ?

Well, there definitely should be some option to select device (or specific window if possible). But what to grab (device, window or whole screen) should be decided by a library user. If such option isn't supported by current platform/configuration a user should see some warning in logs. Any command line arguments should not be omitted by Jaffree since ffmpeg behaviour may change in the future.

More generally speaking, does Jaffree expose a "low level" API that would allow a client app to build its own FFmpeg "recipe" while benefiting from the wrapping layer of Jaffree ?

Not sure what do you mean by low-level API. Is it ProcessHandler? Actually you can add custom Input, Output, ProgressListener and OutputListern and you will get whole control over command line.

…iguration. If not supported, calling setArea does nothing except log a warning with explanations. Save for cursor inclusion option if not supported. Also added Javadoc.

…re level on this OS/configuration.

vicnevicne · 2020-06-11T23:56:55Z

I'll take a look at com.github.kokorin.jaffree.ffmpeg.OutputListener, thanks.

Once device retrieval is done, I think a separate "DeviceCaptureInput" class would be best suited, and maybe DesktopCaptureInput should become a subclass of it.

And yeah, I agree a warning in the logs for unsupported features is the way to go. It's done now, and I also added a query method for user code to determine if the crop feature is supported or not. The example is updated accordingly and chooses between input cropping option and additional crop filter according to the OS/configuration at runtime.

After digging a bit, yes, by "low level", I meant the ProcessHandler indeed. Thanks.

vicnevicne · 2020-06-12T00:01:08Z

By the way, am I correct that Jaffree currently only supports the standard "filter" and "complexfilter" options, but not the "per-stream" filter like '-vf' (shorthand for '-filter:v') and '-af' (or '-filter:a') ?
After testing, it seems 'crop' can be used as a generic '-filter', but all documentation describe it as "video only" and thus use it after '-vf' or '-filter:v'.
If so, I would be tempted to add a 2-parameter version to setFilter like so:

    // Current prototype
    public FFmpeg setFilter(String filter) {
        return setFilter(filter, null);
    }
    // Additional prototype
    public FFmpeg setFilter(String filter, FilterStream filterStream) {
        this.filter = filter;
        this.filterStream = filterStream:
        return this;
    }

where the FilterStream enum could be "AUDIO" or "VIDEO" and, if specified, would replace the "-filter" prefix by "-filter:a" or "-filter:v" ?

What do you think ?
If it sounds good to you, I guess I would make that small change a separate PR.

Kind regards, Vicne

kokorin · 2020-06-13T10:19:19Z

By the way, am I correct that Jaffree currently only supports the standard "filter" and "complexfilter" options, but not the "per-stream" filter like '-vf' (shorthand for '-filter:v') and '-af' (or '-filter:a') ?

Seems so, but Jaffree does allow arbitrary parameters added, so currently there is a workaround.

If so, I would be tempted to add a 2-parameter version to setFilter like so:
    // Current prototype
    public FFmpeg setFilter(String filter) {
        return setFilter(filter, null);
    }
    // Additional prototype
    public FFmpeg setFilter(String filter, FilterStream filterStream) {
        this.filter = filter;
        this.filterStream = filterStream:
        return this;
    }
where the FilterStream enum could be "AUDIO" or "VIDEO" and, if specified, would replace the "-filter" prefix by "-filter:a" or "-filter:v" ?

What do you think ?
If it sounds good to you, I guess I would make that small change a separate PR.

Kind regards, Vicne

Check com.github.kokorin.jaffree.ffmpeg.BaseInOut#setCodec(java.lang.String, java.lang.String). In short there are 2 overloaded methods: the one accepts StreamType enum, the other - streamSpecifier String. Probably it's better to keep such mathods uniform. Also streamSpecifier or StreamType should be the first parameter.

I will happily aceept such PR, please don't forget about JavaDoc (I'm working on it in CHECKSTYLE branch), as well as tests. Also probably there should be more methods which accept FilterGraph.

vicnevicne · 2020-06-14T00:11:37Z

See #80 for "-af" and "-vf" first draft, don't hesitate to comment.

…n in case the capture can't keep up with the requested frame rate

vicnevicne · 2020-06-18T22:46:49Z

Thanks for merging the -filter:v PR. I updated this code to use it.

I'm now working on device list parsing but I have an issue with the OutputListener: it seems it does not get the lines which list all devices (although the LOGGER lists them), but only the first section with ffmpeg version and configuration.

Plus, there is a RuntimeException complaining that ffmpeg exited with a message which, in fact, is the last line of the "version header".

Here is my code:

        FFmpegResult result = FFmpeg.atPath(ffmpegBin)
                .addArgument("-devices")
                .setOutputListener(new OutputListener() {
                    public boolean onOutput(String line) {
                        System.err.println(" PARSING '" + line + "'");
                        return true;
                    }
                })
                .execute();

and here is the output:

INFO main com.github.kokorin.jaffree.process.ProcessHandler - Command constructed:
C:\utils\ffmpeg\bin\ffmpeg -n -devices
INFO main com.github.kokorin.jaffree.process.ProcessHandler - Starting process: C:\utils\ffmpeg\bin\ffmpeg
INFO main com.github.kokorin.jaffree.process.ProcessHandler - Waiting for process to finish
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader - Devices:
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader -  D. = Demuxing supported
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader -  .E = Muxing supported
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader -  --
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader -  D  dshow           DirectShow capture
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader -  D  gdigrab         GDI API Windows frame grabber
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader -  D  lavfi           Libavfilter virtual input device
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader -   E sdl,sdl2        SDL2 output device
INFO StdOut com.github.kokorin.jaffree.process.LoggingStdReader -  D  vfwcap          VfW video capture
INFO main com.github.kokorin.jaffree.process.ProcessHandler - Process has finished with status: 0
WARN StdErr com.github.kokorin.jaffree.process.Executor - Interrupting starter thread (main) because of exception: ffmpeg exited with message:   libpostproc    55.  6.100 / 55.  6.100
WARN main com.github.kokorin.jaffree.process.ProcessHandler - Process has been interrupted
 PARSING 'ffmpeg version git-2020-05-27-8b5ffae Copyright (c) 2000-2020 the FFmpeg developers'
 PARSING '  built with gcc 9.3.1 (GCC) 20200523'
 PARSING '  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf'
 PARSING '  libavutil      56. 49.100 / 56. 49.100'
 PARSING '  libavcodec     58. 87.101 / 58. 87.101'
 PARSING '  libavformat    58. 43.100 / 58. 43.100'
 PARSING '  libavdevice    58.  9.103 / 58.  9.103'
 PARSING '  libavfilter     7. 83.100 /  7. 83.100'
 PARSING '  libswscale      5.  6.101 /  5.  6.101'
 PARSING '  libswresample   3.  6.100 /  3.  6.100'
 PARSING '  libpostproc    55.  6.100 / 55.  6.100'
Exception in thread "main" java.lang.RuntimeException: Failed to execute, exception appeared in one of helper threads
	at com.github.kokorin.jaffree.process.ProcessHandler.interactWithProcess(ProcessHandler.java:128)
	at com.github.kokorin.jaffree.process.ProcessHandler.execute(ProcessHandler.java:85)
	at com.github.kokorin.jaffree.ffmpeg.FFmpeg.execute(FFmpeg.java:234)
	at examples.ffmpeg.ListFeatures.execute(ListFeatures.java:29)
	at examples.ffmpeg.ListFeatures.main(ListFeatures.java:39)
Caused by: java.lang.RuntimeException: Exception during execution
	at com.github.kokorin.jaffree.process.Executor.getException(Executor.java:95)
	at com.github.kokorin.jaffree.process.ProcessHandler.interactWithProcess(ProcessHandler.java:125)
	... 4 more
Caused by: java.lang.RuntimeException: ffmpeg exited with message:   libpostproc    55.  6.100 / 55.  6.100
	at com.github.kokorin.jaffree.ffmpeg.FFmpegResultReader.read(FFmpegResultReader.java:91)
	at com.github.kokorin.jaffree.ffmpeg.FFmpegResultReader.read(FFmpegResultReader.java:32)
	at com.github.kokorin.jaffree.process.ProcessHandler$2.run(ProcessHandler.java:170)
	at com.github.kokorin.jaffree.process.Executor$1.run(Executor.java:67)
	at java.lang.Thread.run(Thread.java:745)

Process finished with exit code 1

Any hint ?

vicnevicne · 2020-06-18T23:06:35Z

I also have a separate issue: when trying to stop a desktop capture using one of the 3 "Forceful stop" methods from the README.md (tried the future.cancel(true) and the exception in ProgressListener), The Java process goes on even though the console shows:

WARN FFmpeg-async-runner com.github.kokorin.jaffree.process.ProcessHandler - Process has been interrupted
WARN FFmpeg-async-runner com.github.kokorin.jaffree.process.Executor - Interrupting ALIVE thread: StdErr
WARN FFmpeg-async-runner com.github.kokorin.jaffree.process.Executor - Interrupting ALIVE thread: StdOut

I then have to kill the Java process but the resulting output file is corrupt (less than 1KB).

When using ffmpeg in command-line, one can exit cleanly by either hitting CTRL-C or just typing the 'q' key.

I'm tempted to think this method would be much cleaner than killing the process, but I see the stdInWriter of the ProcessHandler for FFmpeg is fixed at createStdInWriter() (which returns null).

What would be your advice to be able to send keystrokes to a running FFmpeg instance ?

Kind regards, Vicne

kokorin · 2020-06-19T07:00:33Z

I'm now working on device list parsing but I have an issue with the OutputListener: it seems it does not get the lines which list all devices (although the LOGGER lists them), but only the first section with ffmpeg version and configuration.

ffmpeg prints different data to both stdout & stderr. Seems you need to read not from OutputListener, but to replace StdOutReader or StdErrReader.

I also have a separate issue: when trying to stop a desktop capture using one of the 3 "Forceful stop" methods from the README.md (tried the future.cancel(true) and the exception in ProgressListener), The Java process goes on even though the console shows:

That's definitely caused by force stop of ffmpeg. I haven't experimented with graceful stop. Probably you have to pass q (or any other symbol?) to ffmpeg's input. Check StdInWriter. Probably it can help you. I think graceful stop deserves separate PR.

I want to thank you for your work on Jaffree and improving it. Good job!

vicnevicne · 2020-06-26T10:11:50Z

Oh, and if you intend to wrap up a release, could you please consider merging this PR as well ?
As discussed, it does not do device enumeration and selection, but this is not needed for desktop capture.
I think a more generic "device capture" implementation would be another PR anyway...
Thanks.

kokorin · 2020-06-28T11:33:17Z

I have decided to introduce develop branch to follow simplified gitflow.

Please, close this PR and create another one from the same branch to develop branch.

PS today I unfortunatelly had no time to integrate this changes to release. Will do my best to do it tomorrow.

vicnevicne · 2020-06-28T16:35:37Z

I hope I understood correctly the "create another one from the same branch to develop branch" part.
I've created #88 on the 'develop' branch and I'm closing this one.

vicnevicne added 5 commits June 9, 2020 01:13

Added IS_LINUX and IS_MAC (and slightly changed IS_WINDOWS to follow …

1109449

…Apache commons' logic)

Basic desktop capture Input

b10aeaa

Basic desktop capture example

89d82b1

(typo in comment)

f61a741

Added a way to select a given area of the screen

ce253a0

kokorin requested changes Jun 10, 2020

View reviewed changes

More work on area restriction

0ea3f0d

vicnevicne added 3 commits June 10, 2020 21:46

Added Apache License boilerplate

b690760

(refactoring)

1e9f935

Added a test for DesktopCaptureInput.

16c88e4

Disabling testDesktopCapture() for now with an @ignore annotation

6fbdd49

kokorin requested changes Jun 11, 2020

View reviewed changes

vicnevicne added 3 commits June 12, 2020 01:28

Added a method to check if area cropping is supported on this OS/conf…

ce58a77

…iguration. If not supported, calling setArea does nothing except log a warning with explanations. Save for cursor inclusion option if not supported. Also added Javadoc.

Moved cropping to client code with a filter if not supported at captu…

eca6a57

…re level on this OS/configuration.

Updated license boilerplate

73129a7

vicnevicne added 5 commits June 17, 2020 22:57

Oops, fixed a typo

e204f0e

Framerate must be set on output to achieve a time-accurate result eve…

1adf5f3

…n in case the capture can't keep up with the requested frame rate

Merge remote-tracking branch 'upstream/master'

9aaab15

Specify crop is a video stream filter

60b9b3d

Expose GDI vs DirectShow setting.

edfbe2f

vicnevicne mentioned this pull request Jun 19, 2020

Implement "soft" stop #81

Closed

Merge branch 'master' of https://github.com/kokorin/Jaffree

67cc27c

vicnevicne mentioned this pull request Jun 28, 2020

Desktop Capture Input and example (2) #88

Closed

vicnevicne closed this Jun 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Desktop Capture Input and example #79

Desktop Capture Input and example #79

vicnevicne commented Jun 8, 2020 •

edited

Loading

kokorin left a comment

kokorin Jun 10, 2020

vicnevicne Jun 10, 2020

kokorin Jun 11, 2020

kokorin Jun 11, 2020

vicnevicne commented Jun 10, 2020

vicnevicne commented Jun 10, 2020 •

edited

Loading

kokorin commented Jun 11, 2020

vicnevicne commented Jun 11, 2020

kokorin Jun 11, 2020

vicnevicne Jun 11, 2020

kokorin Jun 11, 2020

kokorin commented Jun 11, 2020

vicnevicne commented Jun 11, 2020 •

edited

Loading

vicnevicne commented Jun 12, 2020

kokorin commented Jun 13, 2020

vicnevicne commented Jun 14, 2020

vicnevicne commented Jun 18, 2020 •

edited

Loading

vicnevicne commented Jun 18, 2020 •

edited

Loading

kokorin commented Jun 19, 2020

vicnevicne commented Jun 26, 2020

kokorin commented Jun 28, 2020 •

edited

Loading

vicnevicne commented Jun 28, 2020

Desktop Capture Input and example #79

Desktop Capture Input and example #79

Conversation

vicnevicne commented Jun 8, 2020 • edited Loading

kokorin left a comment

Choose a reason for hiding this comment

kokorin Jun 10, 2020

Choose a reason for hiding this comment

vicnevicne Jun 10, 2020

Choose a reason for hiding this comment

kokorin Jun 11, 2020

Choose a reason for hiding this comment

kokorin Jun 11, 2020

Choose a reason for hiding this comment

vicnevicne commented Jun 10, 2020

vicnevicne commented Jun 10, 2020 • edited Loading

kokorin commented Jun 11, 2020

vicnevicne commented Jun 11, 2020

kokorin Jun 11, 2020

Choose a reason for hiding this comment

vicnevicne Jun 11, 2020

Choose a reason for hiding this comment

kokorin Jun 11, 2020

Choose a reason for hiding this comment

kokorin commented Jun 11, 2020

vicnevicne commented Jun 11, 2020 • edited Loading

vicnevicne commented Jun 12, 2020

kokorin commented Jun 13, 2020

vicnevicne commented Jun 14, 2020

vicnevicne commented Jun 18, 2020 • edited Loading

vicnevicne commented Jun 18, 2020 • edited Loading

kokorin commented Jun 19, 2020

vicnevicne commented Jun 26, 2020

kokorin commented Jun 28, 2020 • edited Loading

vicnevicne commented Jun 28, 2020

vicnevicne commented Jun 8, 2020 •

edited

Loading

vicnevicne commented Jun 10, 2020 •

edited

Loading

vicnevicne commented Jun 11, 2020 •

edited

Loading

vicnevicne commented Jun 18, 2020 •

edited

Loading

vicnevicne commented Jun 18, 2020 •

edited

Loading

kokorin commented Jun 28, 2020 •

edited

Loading