Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wasm: in_exec_wasi: filter_wasm: Make configurable for wasm heap and stack sizes #9146

Merged

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Jul 31, 2024

Currently, heap and stack sizes for Wasm is fixed and not configurable.
This PR indented to make them configurable.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
    Flush        1
    Daemon       Off
    Log_Level    debug
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Hot_Reload   On
    Grace 5

[INPUT]
    Name dummy
    Tag dummy.locals
    Dummy {"message":"dummy","wasm_float1":1.0,"wasm_float2":100.0,"wasm_int1":1,"wasm_int2":100}

[FILTER]
    Name wasm
    WASM_Path rust_clib_filter.aot
    Function_Name rust_clib_filter
    accessible_paths .

[OUTPUT]
    Name  stdout
    Match *
[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    debug
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name exec_wasi
    Tag  exec.wasi.local
    WASI_path wasi_serde_json.wasm
    Parser wasi
    accessible_paths .

[OUTPUT]
    Name  stdout
    Match *
  • Debug log output from testing the change
Fluent Bit v3.1.5
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/08/09 18:47:23] [ info] Configuration:
[2024/08/09 18:47:23] [ info]  flush time     | 1.000000 seconds
[2024/08/09 18:47:23] [ info]  grace          | 5 seconds
[2024/08/09 18:47:23] [ info]  daemon         | 0
[2024/08/09 18:47:23] [ info] ___________
[2024/08/09 18:47:23] [ info]  inputs:
[2024/08/09 18:47:23] [ info]      dummy
[2024/08/09 18:47:23] [ info] ___________
[2024/08/09 18:47:23] [ info]  filters:
[2024/08/09 18:47:23] [ info]      wasm.0
[2024/08/09 18:47:23] [ info] ___________
[2024/08/09 18:47:23] [ info]  outputs:
[2024/08/09 18:47:23] [ info]      stdout.0
[2024/08/09 18:47:23] [ info] ___________
[2024/08/09 18:47:23] [ info]  collectors:
[2024/08/09 18:47:23] [ info] [fluent bit] version=3.1.5, commit=725640616f, pid=1913275
[2024/08/09 18:47:23] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/08/09 18:47:23] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/08/09 18:47:23] [ info] [cmetrics] version=0.9.3
[2024/08/09 18:47:23] [ info] [ctraces ] version=0.5.3
[2024/08/09 18:47:23] [ info] [input:dummy:dummy.0] initializing
[2024/08/09 18:47:23] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/08/09 18:47:23] [debug] [dummy:dummy.0] created event channels: read=21 write=22
[2024/08/09 18:47:23] [debug] [stdout:stdout.0] created event channels: read=23 write=24
[2024/08/09 18:47:23] [ info] [output:stdout:stdout.0] worker #0 started
[2024/08/09 18:47:23] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/08/09 18:47:23] [ info] [sp] stream processor started
[2024/08/09 18:47:25] [debug] [task] created task=0x7072f0036ba0 id=0 OK
[2024/08/09 18:47:25] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] dummy.locals: [[1723196844.280299405, {}], {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy","wasm_float1":1.0,"wasm_float2":100.0,"wasm_int1":1,"wasm_int2":100}", "tag"=>"dummy.locals", "time"=>"2024-08-09T09:47:24.280299405 +0000"}]
[2024/08/09 18:47:25] [debug] [out flush] cb_destroy coro_id=0
[2024/08/09 18:47:25] [debug] [task] destroy task=0x7072f0036ba0 (task_id=0)
^C[2024/08/09 18:47:25] [engine] caught signal (SIGINT)
[2024/08/09 18:47:25] [debug] [task] created task=0x7072f0038530 id=0 OK
[2024/08/09 18:47:25] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/08/09 18:47:25] [ warn] [engine] service will shutdown in max 5 seconds
[2024/08/09 18:47:25] [ info] [input] pausing dummy.0
[0] dummy.locals: [[1723196845.280252631, {}], {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy","wasm_float1":1.0,"wasm_float2":100.0,"wasm_int1":1,"wasm_int2":100}", "tag"=>"dummy.locals", "time"=>"2024-08-09T09:47:25.280252631 +0000"}]
[2024/08/09 18:47:25] [debug] [out flush] cb_destroy coro_id=1
[2024/08/09 18:47:25] [debug] [task] destroy task=0x7072f0038530 (task_id=0)
[2024/08/09 18:47:26] [ info] [engine] service has stopped (0 pending tasks)
[2024/08/09 18:47:26] [ info] [input] pausing dummy.0
[2024/08/09 18:47:26] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/08/09 18:47:26] [ info] [output:stdout:stdout.0] thread worker #5035 
Fluent Bit v3.1.5
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/08/09 18:48:22] [ info] Configuration:
[2024/08/09 18:48:22] [ info]  flush time     | 1.000000 seconds
[2024/08/09 18:48:22] [ info]  grace          | 5 seconds
[2024/08/09 18:48:22] [ info]  daemon         | 0
[2024/08/09 18:48:22] [ info] ___________
[2024/08/09 18:48:22] [ info]  inputs:
[2024/08/09 18:48:22] [ info]      exec_wasi
[2024/08/09 18:48:22] [ info] ___________
[2024/08/09 18:48:22] [ info]  filters:
[2024/08/09 18:48:22] [ info] ___________
[2024/08/09 18:48:22] [ info]  outputs:
[2024/08/09 18:48:22] [ info]      stdout.0
[2024/08/09 18:48:22] [ info] ___________
[2024/08/09 18:48:22] [ info]  collectors:
[2024/08/09 18:48:22] [ info] [fluent bit] version=3.1.5, commit=725640616f, pid=1913719
[2024/08/09 18:48:22] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/08/09 18:48:22] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/08/09 18:48:22] [ info] [cmetrics] version=0.9.3
[2024/08/09 18:48:22] [ info] [ctraces ] version=0.5.3
[2024/08/09 18:48:22] [ info] [input:exec_wasi:exec_wasi.0] initializing
[2024/08/09 18:48:22] [ info] [input:exec_wasi:exec_wasi.0] storage_strategy='memory' (memory only)
[2024/08/09 18:48:22] [debug] [exec_wasi:exec_wasi.0] created event channels: read=21 write=22
[2024/08/09 18:48:22] [debug] [input:exec_wasi:exec_wasi.0] interval_sec=1 interval_nsec=0 oneshot=0 buf_size=4096
[2024/08/09 18:48:22] [debug] [stdout:stdout.0] created event channels: read=23 write=24
[2024/08/09 18:48:22] [ info] [sp] stream processor started
[2024/08/09 18:48:22] [ info] [output:stdout:stdout.0] worker #0 started
[2024/08/09 18:48:23] [debug] [task] created task=0x7579a007e1c0 id=0 OK
[2024/08/09 18:48:23] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] exec.wasi.local: [[1723196902.306833567, {}], {"age"=>43, "name"=>"John Doe", "phones"=>["+44 1234567", "+44 2345678"]}]
[2024/08/09 18:48:23] [debug] [out flush] cb_destroy coro_id=0
[2024/08/09 18:48:23] [debug] [task] destroy task=0x7579a007e1c0 (task_id=0)
^C[2024/08/09 18:48:24] [engine] caught signal (SIGINT)
[2024/08/09 18:48:24] [debug] [task] created task=0x7579a007fdd0 id=0 OK
[2024/08/09 18:48:24] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/08/09 18:48:24] [ warn] [engine] service will shutdown in max 5 seconds
[2024/08/09 18:48:24] [ info] [input] pausing exec_wasi.0
[0] exec.wasi.local: [[1723196903.291697447, {}], {"age"=>43, "name"=>"John Doe", "phones"=>["+44 1234567", "+44 2345678"]}]
[2024/08/09 18:48:24] [debug] [out flush] cb_destroy coro_id=1
[2024/08/09 18:48:24] [debug] [task] destroy task=0x7579a007fdd0 (task_id=0)
[2024/08/09 18:48:24] [ info] [engine] service has stopped (0 pending tasks)
[2024/08/09 18:48:24] [ info] [input] pausing exec_wasi.0
[2024/08/09 18:48:24] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/08/09 18:48:24] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Update: fluent/fluent-bit-docs#1428

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@edsiper
Copy link
Member

edsiper commented Aug 9, 2024

@cosmo0920 pls add here the PR for docs (cc: @lecaros )

@edsiper edsiper merged commit 68931d1 into master Aug 9, 2024
46 checks passed
@edsiper edsiper deleted the cosmo0920-make-configurable-for-wasm-heap-and-stack-sizes branch August 9, 2024 19:44
@cosmo0920
Copy link
Contributor Author

cosmo0920 commented Aug 13, 2024

I added a link for the doc PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants