in_someip: SOME/IP input plugin #9570

anthonypayne-GM · 2024-11-08T14:44:50Z

This change introduces a new input plugin, in_someip. This plugin is used to subscribe to events and capture notifications from a SOME/IP network. It also can be used in inject SOME/IP requests and capture the response to the data pipeline.

The input plugin uses an open source implementation of the SOME/IP stack, vsomeip. This library is used unchanged, but a wrapper library, libsomeip-c, exposing C-language APIs is part of the PR.

For reference to SOME/IP: protocol specification.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change
Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

[N/A ] Documentation required for this feature

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

The SOME/IP C library is a wrapper around vsomeip. Signed-off-by: Anthony Payne <[email protected]>

Signed-off-by: Anthony Payne <[email protected]>

All tests pass and valgrind does not show any memory errors. Signed-off-by: Anthony Payne <[email protected]>

Signed-off-by: Anthony Payne <[email protected]>

Added code in error legs to free decode buffer. Signed-off-by: Anthony Payne <[email protected]>

Someip input plugin

anthonypayne-GM · 2024-11-08T14:46:24Z

Example config file is included in the PR, conf/in_someip.conf:

[INPUT]
    Name someip
    Tag  in.someip

    # Events to subscribe to.
    # Each event should have form:
    #    Event <service id>,<instance id>,<event id>,<event group 1>,...
    #
    # Each event must have at least one event group
    Event 4,1,32768,1
    Event 4,1,32769,2

    # RPC to send on startup
    # Each RPC entry should have form:
    #    RPC <service id>,<instance id>,<method id>,<Request Payload>
    #
    # Request payload should be base64 encoded
    RPC   4,1,1,CgAQAw==

anthonypayne-GM · 2024-11-08T14:50:20Z

Debug log from runtime tests is attached.
rt_test.log

anthonypayne-GM · 2024-11-08T14:50:59Z

Output from valgrind:

[ OK ]
SUCCESS: All unit tests have passed.
==7910== 
==7910== HEAP SUMMARY:
==7910==     in use at exit: 5,741 bytes in 13 blocks
==7910==   total heap usage: 17,360 allocs, 17,347 frees, 5,723,214 bytes allocated
==7910== 
==7910== LEAK SUMMARY:
==7910==    definitely lost: 0 bytes in 0 blocks
==7910==    indirectly lost: 0 bytes in 0 blocks
==7910==      possibly lost: 0 bytes in 0 blocks
==7910==    still reachable: 5,741 bytes in 13 blocks
==7910==         suppressed: 0 bytes in 0 blocks
==7910== Rerun with --leak-check=full to see details of leaked memory
==7910== 
==7910== For lists of detected and suppressed errors, rerun with: -s
==7910== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

patrick-stephens

Thanks for the contribution, I think we need to look at those runtime dependencies though at the very least - also do we need to include them in the target builds as well under ./packaging/distros?

I don't think we can realistically globally enable this for every supported target. It also seems to use C++ compilation and dependencies.

patrick-stephens · 2024-11-12T14:06:03Z

dockerfiles/Dockerfile

+    libboost-system-dev \
+    libboost-thread-dev \
+    libboost-filesystem-dev \


This is including the dev libraries in the runtime image - we should just have the minimal set of libraries needed for running fluent bit, not those for building it.

Hi @patrick-stephens, I tried to take these out, leaving only the entries above (line 57). But when I did, I was not able to build the SOME/IP input plugin and its dependent libraries from the dev container.

Likewise when I tried to replace these in this section with the "non-dev" version (libboost-filesystem1.74.0) the header files are not installed in the container and the build fails as well.

Please advise if there is something I am doing wrong here.

Yeah I think you need to look at why, the dev libraries should only be in the build stage which is fine. At runtime we need the minimal set of dependencies required to run with, otherwise we end up opening a big bag of security worms with extra dependencies.

Header files are not needed at runtime, they should not be in the runtime image.

So I think you need to ensure the builder stage has the dev libraries to compile everything but then the dep-extractor stage only includes the runtime libraries.

patrick-stephens · 2024-11-12T14:06:18Z

dockerfiles/Dockerfile

@@ -221,6 +227,9 @@ RUN echo "deb http://deb.debian.org/debian bookworm-backports main" >> /etc/apt/
    libatomic1 \
    libgcrypt20 \
    libyaml-0-2 \
+    libboost-system-dev \


As above, do we really need the development libraries here or can we just use the runtime ones?

Same as above, I tried to change this section to the use the non-dev libs and was not able to build the SOME/IP plugin/libs from the dev container.

This section is unrelated to building so changing it here should have no impact on compilation.

Yes but this is not compiling anything, this is the debug image so again should just have the minimum set of packages per the deb-extractor stage.

dockerfiles/Dockerfile

patrick-stephens · 2024-11-12T14:07:58Z

lib/libsomeip-c/CMakeLists.txt

+FetchContent_Declare(
+    vsomeip3
+    GIT_REPOSITORY https://github.com/COVESA/vsomeip
+    GIT_TAG        0b83e24d16e1611958194e9b727136522f46556b # 3.5.1
+)
+FetchContent_MakeAvailable(vsomeip3)


Dependencies are generally vendored into the /lib directory and not fetched remotely.

Thanks. I added the source for vsomeip 3.5.1 as a sub-directory here and updated CMake file to build it before the C wrapper code.

patrick-stephens · 2024-11-12T14:08:38Z

CMakeLists.txt

@@ -227,6 +227,7 @@ if(FLB_ALL)
  set(FLB_IN_NGINX_STATUS 1)
  set(FLB_IN_EXEC         1)
  set(FLB_IN_UNIX_SOCKET  1)
+  set(FLB_IN_SOMEIP       1)


I don't think you can enable this globally without considering all platforms that are supported including Windows and macOS configuration.

Thanks @patrick-stephens , will remove this here.

patrick-stephens · 2024-11-12T14:10:32Z

lib/libsomeip-c/example/test_service.cc

@@ -0,0 +1,138 @@
+/*  Fluent Bit


This seems to be a C++ file, is that just a standard example and/or can it be C only? It seems to require a C++ compiler by looking at the example cmake linkage.

Yes, this is a C++ file.
I used it to execute against fluent-bit with the SOME/IP input plugin enabled.

I will re-write it in C.

patrick-stephens · 2024-11-12T14:14:40Z

For new plugins we must ensure it builds for all targets and CI so I've triggered that. It looks like we have a lot of failures around missing dependencies and the like in various places/platforms.

anthonypayne-GM

Tried to address some of the received comments. Will push modifications to the PR shortly.

Added the vsomeip 3.5.1 source to libsomeip-c library. Updated CMake file to remove the remote fetch and instead build the vsomeip source in sub-directory. Signed-off-by: Anthony Payne <[email protected]>

Re-wrote the test SOME/IP service code in C, rather than in C++. Tested against fluent-bit binary with the in-someip plugin enabled. Signed-off-by: Anthony Payne <[email protected]>

Removed SOME/IP input plugin from the global list in the top level makefile. Signed-off-by: Anthony Payne <[email protected]>

anthonypayne-GM · 2024-12-09T17:03:47Z

Hi @patrick-stephens ,

In order to develop this plugin, I needed an update to the Docker image used by the devcontainer (.devcontainer/devcontainer.json) to include the boost dev libraries.

	"name": "devcontainer-fluent-bit",
	"image": "fluent/fluent-bit:latest-debug",

To get an update to this image, I modified ./dockerfiles/Dockerfile as per the PR and locally installed the image with the boost dev libraries ($ docker build -t fluent/fluent-bit:latest-debug -f dockerfiles/Dockerfile .).

The dev-container uses the docker image to build fluent bit so that is why I need the dev versions of the boost library in the container image.

It appears from the comments though, that the Docker file I updated in the PR is not the correct one for updating the docker image used by the dev container?

Can you please provide some guidance on how I should proceed?

Also, I am not sure about the Docker files that are under packaging/distros. Should those be updated with the plugin dependencies as well?

anthonypayne-GM · 2024-12-09T23:00:13Z

For new plugins we must ensure it builds for all targets and CI so I've triggered that. It looks like we have a lot of failures around missing dependencies and the like in various places/platforms.

This plugin should only be built for linux environments (no MacOS or Windows). Since this plugin requires boost to be installed on Ubuntu do I need to update the .github/workflows/unit-tests.yaml file to add in the apt install of the required boost libraries?

For Centos 7, the available boost from yum repo is to old to support this plugin (vsomeip requires boost 1.66 or newer; yum repo has boost 1.53 I think).

anthonypayne-GM added 8 commits October 28, 2024 11:12

lib: someip-c: Added SOME/IP C library

98c94e6

The SOME/IP C library is a wrapper around vsomeip. Signed-off-by: Anthony Payne <[email protected]>

in_someip: Added SOME/IP input plugin

b1e9803

Signed-off-by: Anthony Payne <[email protected]>

in_someip: Added runtime tests for in_someip

a7e41f4

All tests pass and valgrind does not show any memory errors. Signed-off-by: Anthony Payne <[email protected]>

in_someip: Added sample config file for in_someip

943391b

Signed-off-by: Anthony Payne <[email protected]>

in_someip: Clean up plugin from PR review comments

a2b46b3

Signed-off-by: Anthony Payne <[email protected]>

lib: in_someip: Clean up from PR review comments

0142e58

Signed-off-by: Anthony Payne <[email protected]>

in_someip: Clean up memory leak

ded0e49

Added code in error legs to free decode buffer. Signed-off-by: Anthony Payne <[email protected]>

Merge pull request #4 from anthonypayne-GM/someip-input-plugin

6763dde

Someip input plugin

anthonypayne-GM requested review from niedbalski, patrick-stephens, celalettin1286, fujimotos, edsiper, leonardo-albertovich and koleini as code owners November 8, 2024 14:44

github-actions bot added the docs-required label Nov 8, 2024

patrick-stephens requested changes Nov 12, 2024

View reviewed changes

patrick-stephens added the ok-package-test Run PR packaging tests label Nov 12, 2024

anthonypayne-GM temporarily deployed to pr November 12, 2024 14:12 — with GitHub Actions Inactive

anthonypayne-GM temporarily deployed to pr November 12, 2024 14:36 — with GitHub Actions Inactive

anthonypayne-GM temporarily deployed to pr November 12, 2024 14:37 — with GitHub Actions Inactive

anthonypayne-GM commented Nov 18, 2024

View reviewed changes

lib: someip-c: Added in vsomeip 3.5.1 source

3905bf5

Added the vsomeip 3.5.1 source to libsomeip-c library. Updated CMake file to remove the remote fetch and instead build the vsomeip source in sub-directory. Signed-off-by: Anthony Payne <[email protected]>

anthonypayne-GM added 2 commits November 18, 2024 13:02

lib: someip-c: Re-wrote test service in C

3be318c

Re-wrote the test SOME/IP service code in C, rather than in C++. Tested against fluent-bit binary with the in-someip plugin enabled. Signed-off-by: Anthony Payne <[email protected]>

in_someip: Update global plugin list

239b46b

Removed SOME/IP input plugin from the global list in the top level makefile. Signed-off-by: Anthony Payne <[email protected]>

anthonypayne-GM changed the title ~~SOME/IP input plugin~~ in_someip: SOME/IP input plugin Dec 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

in_someip: SOME/IP input plugin #9570

in_someip: SOME/IP input plugin #9570

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

patrick-stephens left a comment

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens Nov 18, 2024 •

edited

Loading

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens Nov 18, 2024

patrick-stephens Nov 18, 2024

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens commented Nov 12, 2024

anthonypayne-GM left a comment

anthonypayne-GM commented Dec 9, 2024

anthonypayne-GM commented Dec 9, 2024

in_someip: SOME/IP input plugin #9570

Are you sure you want to change the base?

in_someip: SOME/IP input plugin #9570

Conversation

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

patrick-stephens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrick-stephens Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrick-stephens commented Nov 12, 2024

anthonypayne-GM left a comment

Choose a reason for hiding this comment

anthonypayne-GM commented Dec 9, 2024

anthonypayne-GM commented Dec 9, 2024

patrick-stephens Nov 18, 2024 •

edited

Loading