Support for both starts_when and ends_when in reduce transform

[ssboisen]

When starts_when option for the reduce transform were introduced in #4771 it's mentioned by @JeanMertz that in the future the restriction for specifying both that and ends_with at ones could be lifted if a use-case were found. I believe I have one.

The situation is as follows, I have log lines with the following messages where each line is a separate log event:

line1
##[startoftransactionmarker]line 2
line 3
line 4
##[endoftransactionmarker]
line 5

From those 6 events I would after the reductions have the following 3 events separated by blank lines

line1

##[startoftransactionmarker]line 2
line 3
line 4
##[endoftransactionmarker]

line 5

My initial thought was to just have the following configuration:

[transforms.reduce]
type = "reduce"
inputs = [ "my-input" ]
merge_strategies.message = "concat_newline"
starts_when = """starts_with!(.message, "##[startoftransactionmarker]")"""
ends_when = """starts_with!(.message, "##[endoftransactionmarker]")"""

However I found that it's currently not possible due to the restriction mentioned above resulting in an error when starting vector. Does anyone have a suggestion how you would do this using reduce without using both starts_when and ends_when? Currently I have resorted to using lua transform instead.

I have created the following behavior-test that passes when the 3 lines that restricts usage of starts_when and ends_when are removed.

[transforms.starts_when_and_ends_when]
    inputs = []
    type = "reduce"
    merge_strategies.message = "concat_newline"
    starts_when.type = "vrl"
    starts_when.source = """starts_with!(.message, "##[startoftransactionmarker]")"""
    ends_when.type = "vrl"
    ends_when.source = """starts_with!(.message, "##[endoftransactionmarker]")"""

[[tests]]
  name = "reduce_starts_when_and_ends_when"

  [[tests.inputs]]
    insert_at = "starts_when_and_ends_when"
    type = "log"
    [tests.inputs.log_fields]
      message = "line 1"

  [[tests.inputs]]
    insert_at = "starts_when_and_ends_when"
    type = "log"
    [tests.inputs.log_fields]
      message = "##[startoftransactionmarker]line 2"

  [[tests.inputs]]
    insert_at = "starts_when_and_ends_when"
    type = "log"
    [tests.inputs.log_fields]
      message = "line 3"

  [[tests.inputs]]
    insert_at = "starts_when_and_ends_when"
    type = "log"
    [tests.inputs.log_fields]
      message = "line 4"

  [[tests.inputs]]
    insert_at = "starts_when_and_ends_when"
    type = "log"
    [tests.inputs.log_fields]
      message = "##[endoftransactionmarker]"

  [[tests.inputs]]
    insert_at = "starts_when_and_ends_when"
    type = "log"
    [tests.inputs.log_fields]
      message = "line 5"

  [[tests.outputs]]
    extract_from = "starts_when_and_ends_when"
    [[tests.outputs.conditions]]
      type = "vrl"
      source = '''
        message = string!(.message)
        message == "line 1"
      '''
    [[tests.outputs.conditions]]
      type = "vrl"
      source = '''
        message = string!(.message)
        message == "##[startoftransactionmarker]line 2\nline 3\nline 4\n##[endoftransactionmarker]"
      '''
    [[tests.outputs.conditions]]
      type = "vrl"
      source = '''
        message = string!(.message)
        message == "line 5"
      '''

[szibis]

Yes, I have a similar case to have both at the same time. Also would be great to use multiple strategies for specific JSON nested fields at the same time on this reduced between start and end.

[ArtemTraierLotus]

@ssboisen how did u bypass that limitation ?