[Page Add][Enhanced Function]Support the camera interface for XIAO ESP32S3 Sense, by Zephyr

[MatthewJeffson]

Overview

• We need your help to support the camera interface for ESP32 device, and support XIAO ESP32S3 Sense at the following part, using Zephyr.
• Reference by Tim, the Seeed Contributor:
  • There's an open issue for further extending zephyr support for the camera and microphone on the device tree.
    boards: Extend XIAO ESPS3 board to support for XIAO ESP32S3 Sense zephyrproject-rtos/zephyr#60148 May be worth circling back here once someone has a chance to tackle that issue to further augment it.

How do we work together?

• Leave a comment with your GitHub name below as you accept this assignment and will complete the contents about what we are trying to present.
• The maintainer will assign and record you as our only contact for this assignment.
• You can first Deploy this Wiki Platform Locally (The entire source code .zip file is only 30 M)
• You will be invited as our "Contributor" once you "Pull Request".
• The maintainer will review the PR and then merge.

What do we offer?

We want to express our gratitude to our contributors by offering a range of rewards. Depending on the difficulties(Tier 0/1/2/3), the time we expect(Days 1/3/7/15), the actual submission time and the actual submission contents, we will offer different levels of rewards.

1. We will provide our products or cash bonus to our contributors as a token of appreciation.
2. For those contributors who join our "Ranger Program"(announced soon), we offer even more exciting rewards, including access to our prototype products.
3. In addition, all contributors and their contributions will be acknowledged and showcased on our wiki platform.

FAQ

• For any question, the contributor can directly commend here.

I think I can complete the assignment, but I need more time?

• The timeframe here is based on our old experience, from people who has done it or something like that.
• However, many projects will take much more time under actual circumstances (including the main language of the contributors receiving the task and the actual creative difficulty)
• You can leave the time you need and a brief reason, we will evaluate the situation accordingly.

[Priyanshu0901]

@MatthewJeffson
I would love to take up on this project.

[MatthewJeffson]

NICE, and thank you! @Priyanshu0901

[havoc-dlt]

Any progress on this one?
I am kinda torn between waiting for it and trying something myself right now.

[Priyanshu0901]

Yes, still a work in progress (still a bit of testing and documentation is left). ETA 1week.

[havoc-dlt]

Thanks a lot for your effort!

[epc-ake]

Hi, I'm also interested in this. Do you have already something to share?

[MatthewJeffson]

Hi! @epc-ake Yes! Thank you for considering it! @Priyanshu0901 Roy is currently not available on this assignment.
If you can help us that would be really great!
Here is my Discord account: matthew_yu0310 and we can discuss more on the Discord.
Best Regards,
Matthew

[bienxanh1901]

Hello Team,
Any progress on this task?
I can only use Xiao ESP32S3 Sense SD card and Micro on Zephyr.
Can you share something about working with camera interface?
Can we just only define device tree, or need further work?

Thank you,
Hai Pham

[epc-ake]

The problem is that there is currently no driver for the lcd_cam component of the esp32s3.
I'm actually writing one now and i already was able to receive some images. However, there are also changes to the DMA driver needed. For those I made a PR yesterday.
Once they are merged I'll continue improving the camera driver and make a PR for it as well.

[MatthewJeffson]

Wow! Thank you for the update!

[bienxanh1901]

The problem is that there is currently no driver for the lcd_cam component of the esp32s3. I'm actually writing one now and i already was able to receive some images. However, there are also changes to the DMA driver needed. For those I made a PR yesterday. Once they are merged I'll continue improving the camera driver and make a PR for it as well.

Thank you so much for your hard working @epc-ake!

[epc-ake]

Hi all,

I made two drafts for a pull requests: zephyrproject-rtos/zephyr#75331 and zephyrproject-rtos/zephyr#74974. I can also provide the overlay file needed for the xiao_esp32s3, but I'm not sure where to place it yet.
Note the limitations on external memory mentioned in the PR.

Feel free to test it out and leave your comments.

[MatthewJeffson]

Thank you so much! @epc-ake
We will check it very soon!
Best Regards,
Matthew

[pblxptr]

@epc-ake how did it end up? I saw that support for ESP32S3 CAM interface has been merged to mainline, as well as the PR that allows to allocate video buffer in SPIRAM. How you tried to run these changes with xiao_esp32s3? Possibly with streaming cam over WIFI? I tried myself but did not manage to get any sensible results.

[epc-ake]

HI @pblxptr
Yes we made good progress!
As you mentioned, the video driver has been merged. To be able to capture in full resolution this PR is needed. Looks like this is merged soon too.
But there is also this PR which updates sample/drivers/video/capture and lets you chose a lower resolution for the video stream that fits into internal ram.
After that we probably need to add a conf file for the xiao_esp32 board to the sample to make it work out of the box.
However, with the right configs set, I'm able to capture video in full resolution now.
I also managed to stream video over wifi, but only with a hacky application I did myself.

To be able to run the sample/drivers/video/capture sample, try the following:

1. check out newest zephyr main
2. apply (this PR )
3. apply this PR
4. then copy the samples/drivers/video/capture/boards/esp32s3_eye_procpu.conf to a new file named samples/drivers/video/capture/boards/xiao_esp32s3_procpu.conf
5. Build and run the sample/drivers/video/capture sample

[pblxptr]

HI @pblxptr Yes we made good progress! As you mentioned, the video driver has been merged. To be able to capture in full resolution this PR is needed. Looks like this is merged soon too. But there is also this PR which updates sample/drivers/video/capture and lets you chose a lower resolution for the video stream that fits into internal ram. After that we probably need to add a conf file for the xiao_esp32 board to the sample to make it work out of the box. However, with the right configs set, I'm able to capture video in full resolution now. I also managed to stream video over wifi, but only with a hacky application I did myself.

To be able to run the sample/drivers/video/capture sample, try the following:

1. check out newest zephyr main

2. apply ([this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-16cf59216520b76034171d2c94fc72dc3bdcf0af722255a2c226c58cd35de6df) )

3. apply [this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-737bb0ca878e20bfaaf493aae9eb42d3de0e40e22a0fd17ac08d826050c1d41b)

4. then copy the `samples/drivers/video/capture/boards/esp32s3_eye_procpu.conf` to a new file named `samples/drivers/video/capture/boards/xiao_esp32s3_procpu.conf`

5. Build and run the `sample/drivers/video/capture` sample

Hi thanks, basically I've already done everything what you just suggested and it looks good, however I'm struggling with sending the stream over wifi (I guess there are some issues with sharing SPIRAM between video and wifi, anyway I keep tracking some ongoing PRs with WiFi changes, mostly related to heap and SPIRAM), perhaps I also need to make some more hacks in my app.

[epc-ake]

HI @pblxptr Yes we made good progress! As you mentioned, the video driver has been merged. To be able to capture in full resolution this PR is needed. Looks like this is merged soon too. But there is also this PR which updates sample/drivers/video/capture and lets you chose a lower resolution for the video stream that fits into internal ram. After that we probably need to add a conf file for the xiao_esp32 board to the sample to make it work out of the box. However, with the right configs set, I'm able to capture video in full resolution now. I also managed to stream video over wifi, but only with a hacky application I did myself.
To be able to run the sample/drivers/video/capture sample, try the following:

1. check out newest zephyr main

2. apply ([this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-16cf59216520b76034171d2c94fc72dc3bdcf0af722255a2c226c58cd35de6df) )

3. apply [this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-737bb0ca878e20bfaaf493aae9eb42d3de0e40e22a0fd17ac08d826050c1d41b)

4. then copy the `samples/drivers/video/capture/boards/esp32s3_eye_procpu.conf` to a new file named `samples/drivers/video/capture/boards/xiao_esp32s3_procpu.conf`

5. Build and run the `sample/drivers/video/capture` sample

Hi thanks, basically I've already done everything what you just suggested and it looks good, however I'm struggling with sending the stream over wifi (I guess there are some issues with sharing SPIRAM between video and wifi, anyway I keep tracking some ongoing PRs with WiFi changes, mostly related to heap and SPIRAM), perhaps I also need to make some more hacks in my app.

Okay so you are able to run the sample/driver/video/capture sample?
I have to admit, I've never tried to stream over WIFI from external ram. Maybe there are some problems I didn't run over jet. But I was able to stream from internal ram without any problems.

[pblxptr]

HI @pblxptr Yes we made good progress! As you mentioned, the video driver has been merged. To be able to capture in full resolution this PR is needed. Looks like this is merged soon too. But there is also this PR which updates sample/drivers/video/capture and lets you chose a lower resolution for the video stream that fits into internal ram. After that we probably need to add a conf file for the xiao_esp32 board to the sample to make it work out of the box. However, with the right configs set, I'm able to capture video in full resolution now. I also managed to stream video over wifi, but only with a hacky application I did myself.
To be able to run the sample/drivers/video/capture sample, try the following:

1. check out newest zephyr main

2. apply ([this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-16cf59216520b76034171d2c94fc72dc3bdcf0af722255a2c226c58cd35de6df) )

3. apply [this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-737bb0ca878e20bfaaf493aae9eb42d3de0e40e22a0fd17ac08d826050c1d41b)

4. then copy the `samples/drivers/video/capture/boards/esp32s3_eye_procpu.conf` to a new file named `samples/drivers/video/capture/boards/xiao_esp32s3_procpu.conf`

5. Build and run the `sample/drivers/video/capture` sample

Hi thanks, basically I've already done everything what you just suggested and it looks good, however I'm struggling with sending the stream over wifi (I guess there are some issues with sharing SPIRAM between video and wifi, anyway I keep tracking some ongoing PRs with WiFi changes, mostly related to heap and SPIRAM), perhaps I also need to make some more hacks in my app.

Okay so you are able to run the sample/driver/video/capture sample? I have to admit, I've never tried to stream over WIFI from external ram. Maybe there are some problems I didn't run over jet. But I was able to stream from internal ram without any problems.

Yep I was able to run this sample. To be honest I've stumbled upon your changes after I did mine, because I was looking how to combine all these things together namely wifi and camera using external ram. I will try to hack my app and put everything into internal ram and I'll see what will be the result.

[MatthewJeffson]

Wow! @epc-ake That sounds really amazing! Sorry our guys were busy on the Watcher and I was thinking the new framework of the Contributor Program. It is great to see the progress going well! Happy Mid-Autumn Festival to you guys!
Best Regards,
Matthew

[github-actions]

✅ @MatthewJeffson

This issue is closed, If you have any questions, you can comment and reply.

[pblxptr]

@MatthewJeffson why is this closed? Is the support for XIAO ESP32S3 Sense added to Zephyr?

[MatthewJeffson]

Oh! @pblxptr I am sorry I am looking the wrong issue.

Thanks for the notices!

[pblxptr]

@MatthewJeffson hey I submitted a PR with support for the Xiao ESP32 S3 sense. Any remarks will be highly appreciated.
zephyrproject-rtos/zephyr#79334

[MatthewJeffson]

Oh Hi! @pblxptr Patryk thanks for the PR! I really appreciated! Are you using XIAO ESP32S3 Sense enable all Zephyr functions? including the camera usage?
Best Regards,
Matthew

[pblxptr]

Oh Hi! @pblxptr Patryk thanks for the PR! I really appreciated! Are you using XIAO ESP32S3 Sense enable all Zephyr functions? including the camera usage? Best Regards, Matthew

Hi, enabling camera was my main focus, so camera and sd card available on XIAO ESP32S3 Sense are working. There is also a mic on this board as far as I know, but tbh I didn't need it so I haven't touched it.

[MatthewJeffson]

That is nice! I will have my colleagues contacting the Zephyr Official regarding the PR. By the way, would you mind upgrading the wiki about how to use it? https://wiki.seeedstudio.com/xiao_esp32s3_zephyr_rtos/

Refer to the Contributor Program, I really want to make some compensation for your efforts!

Best Regards,
Matthew

[pblxptr]

That is nice! I will have my colleagues contacting the Zephyr Official regarding the PR. By the way, would you mind upgrading the wiki about how to use it? https://wiki.seeedstudio.com/xiao_esp32s3_zephyr_rtos/

Refer to the Contributor Program, I really want to make some compensation for your efforts!

Best Regards,
Matthew

Sure, once they merge my changes I'll update the wiki

[MatthewJeffson]

NICE! That would be great! Thanks!

[MatthewJeffson]

I think this one can be closed.

[github-actions]

✅ @MatthewJeffson

This issue is closed, If you have any questions, you can comment and reply.