drivers: video: shell: initial implementation #82393

josuah · 2024-12-02T07:38:23Z

Dependencies:

This is a draft of a video shell, a toolbox to operate video devices interactively for debug and troubleshooting purposes.

It is based on the definition of a single buffer, that is implicitly referred by all commands working on buffers.

It currently the alloc, format, frmimval, enqueue/dequeue, start/stop families of API functions.
It uses the newly introduced with DEVICE_API_IS(video, dev) to avoid crashing when a non-matching API is used.
The sample runs on the native_sim platform using the emulated video drivers
It has extra commands to hexdump and preview a buffer on the terminal.

Example session:

For now, only previewing RGB565 is supported for the preview.

If the buffer is interpreted with the wrong pitch, so it is shown in diagonal lines (i.e. troubleshooting the pitch):

This makes a lower-level debug tap to debug video devices than UVC

josuah · 2024-12-02T07:42:58Z

uart:~$ video --help
video - Video driver commands
Subcommands:
  start      : Start a video device and its sources
               Usage: video start <device>
  stop       : Stop a video device and its sources
               Usage: video stop <device>
  show       : Show video device information
               Usage: video show <device>
  format     : Set the active format of a video device
               Usage: video format <device> <fourcc> <width>x<height>
  framerate  : Set the frame rate of a video device
               Usage: video framerate <device> <fps>
  alloc      : Allocate a buffer to enqueue to video devices, or show the
               current buffer size
               Set to 0 to free the buffer completely, ommit argument to show
               the size
               Usage: video alloc [<size>]
  enqueue    : Send a video buffer to a video device
               Usage: video enqueue <device>
  dump       : Dump the given size (or everythiing) of the buffer at given
               offset (or zero)
               Usage: video dump [-o <offset>] [-s <size>]
  view       : Display the video buffer content on the terminal
               Usage: video view <fourcc> <width> [-o <offset>] [-s <size>] [-c
               <columns>]

uart:~$

kartben · 2024-12-02T10:19:10Z

🤯

kartben · 2024-12-02T14:20:23Z

drivers/video/Kconfig.shell

+config VIDEO_SHELL
+	bool "Video shell commands support"
+	depends on SHELL
+	default y


we typically don't enable shell modules by default

kartben · 2024-12-02T14:42:53Z

drivers/video/video_shell.c

+		return -ENODEV;
+	}
+
+	if (!DEVICE_API_IS(video, dev)) {


since this depends on #82301, maybe include the associated commit in your PR for now?

Yes thank you, this would allow the SW_GENERATOR device to work.

ngphibang · 2024-12-02T14:43:12Z

Hi @josuah, this looks promising. I didn't look into details yet but some of my quick ideas / observations:

IMHO, the demonstration of each video API seems not really helpful ?. Instead, it's better to do real streaming applications ?
Maybe I am wrong, the display part on the console seems working only for test pattern with low resolution ? I don't think we can display real captured image with high resolution via uart due to the its limited bandwidth. So it would be more useful if we display video frames on real LCD rather than onto the console ?
Maybe you are already aware of, there is YAVTA (Yet Another V4L2 Test Application) which is a very useful tool and widely used for video testing in Linux
https://wiki.st.com/stm32mpu/wiki/Yavta#:~:text=Yavta%20stands%20for%20%22Yet%20Another,debugging%20and%20controlling%20camera%20subsystem.
https://git.ideasonboard.org/yavta.git/tree

I am working on another shell utility which looks like v4l2-ctl and media-ctl to inspect / build the pipeline and video device's properties especially the controls part. There may be a small overlapping part with this but it's not a problem (yavta / v4l2-ctl / media-ctl do have some small overlaps)

josuah · 2024-12-02T17:46:48Z

Thank you for your feedback!

IMO, the demonstration of each video API seems not really helpful.

The goal was to bring some preview as early as possible in the development: only RAM and an UART are needed.

Instead, we should focus on real usecase / application. This mean this shell utility should do real streaming and display

This is the way forward! I think could not really realize it: a test utility needs to be able to stream, work on full pipelines rather than individual devices.

Maybe I am wrong, the display part on the console seems working only for test pattern with low resolution ?

It does not do any interpolation, but it does skip pixels, working one a per-line basis.

All Pixels         Pixels Kept      __
::::::::::::::::   . . . . . . . .    \
::::::::::::::::   . . . . . . . .     \__  Packed into a smaller frame
::::::::::::::::   . . . . . . . .          ::::::::
::::::::::::::::   . . . . . . . .  ______  ::::::::

-> I think O(1) as variable number of pixels are skipped, but a constant amount of data is converted.
-> Absolutely terrible-looking image due to the poor man's rescaling used (see how much aliasing below)

FYI, there is YAVTA (from Laurent Pinchart) which is well-known in Linux. I think it would be very helpful if this utility can do something similar

I did not know it! 👍

I am working on another shell utility which looks like v4l2-ctl and media-ctl which allows to inspect / build the pipeline and the devices properties especially the controls part. There will be some small parts overlapping with this but no problem (yavta / v4l2-ctl / media-ctl have also some small overlapping parts)

This PR #82393 helps early driver development: getting the first frame working, and follow it manually through a pipeline. Once a pipeline works, #82393 stops being helpful, something is missing to it!

I will keep it as a draft to avoid getting it in the way of your version, and adapt it as needed, remove the parts that overlap or considered not useful.

ngphibang · 2024-12-02T18:00:15Z

This PR #82393 helps early driver development: getting the first frame working, and follow it manually through a pipeline. Once a pipeline works, #82393 stops being helpful, something is missing to it!

I will keep it as a draft to avoid getting it in the way of your version, and adapt it as needed, remove the parts that overlap or considered not useful.

It's helpful. v4l2-ctl is mainly to set/get_ctrl and inspect some device's infos info while media-ctl is used to inpect the pipeline topology and to build the pipeline. Both tools do not cover streaming, test, debug. I mentionned yavta just to say that this PR is potentially a shell tool to test / debug / stream video frames as yavta does. Sorry if my comment got confused.

josuah · 2024-12-02T18:38:55Z

No confusion :) I mean I am not rushing this, and better leave unimplemented the stub commands if more comprehensive versions would come in another PR.

What yavta does gives me some idea...

0 (0) [-] none 0 460800 B 189819.407102 189819.495226 5.431 fps ts mono/SoE
1 (1) [-] none 1 460800 B 189819.495083 189819.555251 11.366 fps ts mono/SoE
2 (2) [-] none 2 460800 B 189819.555102 189819.615259 16.661 fps ts mono/SoE
3 (3) [-] none 3 460800 B 189819.615108 189819.675216 16.665 fps ts mono/SoE
4 (4) [-] none 4 460800 B 189819.675081 189819.735261 16.674 fps ts mono/SoE
5 (5) [-] none 5 460800 B 189819.735110 189819.795241 16.659 fps ts mono/SoE
6 (6) [-] none 6 460800 B 189819.795098 189819.855151 16.670 fps ts mono/SoE
7 (7) [-] none 7 460800 B 189819.855012 189819.915283 16.691 fps ts mono/SoE

josuah · 2024-12-02T21:49:28Z

force-push:

some CI fixes
cherry-pick to get pdgent/linker-api-remaining

The latter enables the software generator to work, I can remove the emulated drivers as a requirement if that helps

kartben · 2024-12-03T12:36:21Z

drivers/video/video_shell.c

+
+static void dcmd_video_device_name(size_t idx, struct shell_static_entry *entry)
+{
+	const struct device *dev = shell_device_lookup(idx, NULL);


you can make this smarter and only display video devices :)

zephyr/drivers/led/led_shell.c

Lines 332 to 341 in 84532b2

static bool device_is_led_and_ready(const struct device *dev)

{

return device_is_ready(dev) && DEVICE_API_IS(led, dev);

}

static void device_name_get(size_t idx, struct shell_static_entry *entry)

{

const struct device *dev = shell_device_filter(idx, device_is_led_and_ready);

Nice!

uart:~$ video start VIDEO_SW_GENERATOR video_emul_imager video_emul_rx uart:~$ video start

Introduce a video_get_format_index() utility to help finding a caps entry out of a given format. Introduce several utilities to seek and apply frame intervals. Signed-off-by: Josuah Demangeon <[email protected]>

Add a new implementation of a test pattern generator, with the same architecture as real drivers: split receiver core and I2C-controlled sub-device, with changes of video format in "zephyr,emul-imager" leads to different data produced by "zephyr,emul-rx". Signed-off-by: Josuah Demangeon <[email protected]>

Introduction of a video shell command, with subcommands to operate video APIs on a basic level, using a single buffer th be passed to a video device of choice and dumped as hexadecimal or ANSI colors. Signed-off-by: Josuah Demangeon <[email protected]>

josuah · 2024-12-03T14:29:37Z

force-push:

Rebase on top of drivers: Place API in iterable section for remaining drivers #82301
Add filtering for tab-completion to only select video devices

josuah added area: Drivers area: Shell Shell subsystem area: Video Video subsystem labels Dec 2, 2024

josuah marked this pull request as draft December 2, 2024 07:38

zephyrbot added area: Samples Samples area: Devicetree labels Dec 2, 2024

zephyrbot requested review from decsny, galak, kartben, loicpoulain, nashif and ngphibang December 2, 2024 07:39

zephyrbot assigned kartben Dec 2, 2024

kartben reviewed Dec 2, 2024

View reviewed changes

josuah force-pushed the pr-video-shell branch from 8af905d to aa38101 Compare December 2, 2024 14:34

kartben reviewed Dec 2, 2024

View reviewed changes

pdgendt self-requested a review December 2, 2024 15:24

josuah force-pushed the pr-video-shell branch from aa38101 to 468f1f4 Compare December 2, 2024 21:40

kartben reviewed Dec 3, 2024

View reviewed changes

josuah added 2 commits December 3, 2024 12:44

drivers: video: common: Add utilities to seek frmival/caps structures

cf1100d

Introduce a video_get_format_index() utility to help finding a caps entry out of a given format. Introduce several utilities to seek and apply frame intervals. Signed-off-by: Josuah Demangeon <[email protected]>

josuah force-pushed the pr-video-shell branch from 468f1f4 to e2e76fa Compare December 3, 2024 14:25

josuah force-pushed the pr-video-shell branch from e2e76fa to 5e3db41 Compare December 3, 2024 14:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

drivers: video: shell: initial implementation #82393

drivers: video: shell: initial implementation #82393

josuah commented Dec 2, 2024 •

edited

Loading

josuah commented Dec 2, 2024 •

edited

Loading

kartben commented Dec 2, 2024

kartben Dec 2, 2024

kartben Dec 2, 2024

josuah Dec 2, 2024

ngphibang commented Dec 2, 2024 •

edited

Loading

josuah commented Dec 2, 2024 •

edited

Loading

ngphibang commented Dec 2, 2024 •

edited

Loading

josuah commented Dec 2, 2024 •

edited

Loading

josuah commented Dec 2, 2024 •

edited

Loading

kartben Dec 3, 2024

josuah Dec 3, 2024

josuah commented Dec 3, 2024


	static bool device_is_led_and_ready(const struct device *dev)
	{
	return device_is_ready(dev) && DEVICE_API_IS(led, dev);
	}

	static void device_name_get(size_t idx, struct shell_static_entry *entry)
	{
	const struct device *dev = shell_device_filter(idx, device_is_led_and_ready);

drivers: video: shell: initial implementation #82393

Are you sure you want to change the base?

drivers: video: shell: initial implementation #82393

Conversation

josuah commented Dec 2, 2024 • edited Loading

josuah commented Dec 2, 2024 • edited Loading

kartben commented Dec 2, 2024

kartben Dec 2, 2024

Choose a reason for hiding this comment

kartben Dec 2, 2024

Choose a reason for hiding this comment

josuah Dec 2, 2024

Choose a reason for hiding this comment

ngphibang commented Dec 2, 2024 • edited Loading

josuah commented Dec 2, 2024 • edited Loading

ngphibang commented Dec 2, 2024 • edited Loading

josuah commented Dec 2, 2024 • edited Loading

josuah commented Dec 2, 2024 • edited Loading

kartben Dec 3, 2024

Choose a reason for hiding this comment

josuah Dec 3, 2024

Choose a reason for hiding this comment

josuah commented Dec 3, 2024

josuah commented Dec 2, 2024 •

edited

Loading

josuah commented Dec 2, 2024 •

edited

Loading

ngphibang commented Dec 2, 2024 •

edited

Loading

josuah commented Dec 2, 2024 •

edited

Loading

ngphibang commented Dec 2, 2024 •

edited

Loading

josuah commented Dec 2, 2024 •

edited

Loading

josuah commented Dec 2, 2024 •

edited

Loading