[New Integration] Goflow2 integration

[HaveSec]

The GoFlow2 integration allows you to monitor goflow2 logs. At the moment only goflow2 sflow logs are supported.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

• elastic-package check
• elastic-package install
• install elatic-agent
• generate goflow2 sflow logs
  • goflow2 -format json -listen "sflow://:6343" -mapping /root/sflow/goflow2/mapping.yaml -transport.file /var/log/sflow/goflow2/goflow2.log

# File: mapping.yaml
formatter:
    fields: # list of fields to format in JSON
        - type
        - time_flow_start_ns
        - sampler_address
        - sequence_num
        - in_if
        - out_if
        - src_addr
        - dst_addr
        - etype
        - proto
        - src_port
        - dst_port
        - src_vlan
        - dst_vlan
        - ip_flags
        - sampling_rate
        - bytes

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[andrewkroh]

/test

[HaveSec]

/test

[pkoutsovasilis]

this one should be @elastic/sec-deployment-and-devices

[marioschaefer]

fixed.

[marioschaefer]

fixed in 43c25c7 and d737ad8

[pkoutsovasilis]

@HaveSec @marioschaefer could you please add some tests for the sflow datastream?

[marioschaefer]

@HaveSec @marioschaefer could you please add some tests for the sflow datastream?

on my list for today.

[marioschaefer]

@HaveSec @marioschaefer could you please add some tests for the sflow datastream?

on my list for today.

@pkoutsovasilis took a bit longer, but now finished. all tests pass.

[taylor-swanson]

/test

[marioschaefer]

/test

@taylor-swanson codeowner fixed

[taylor-swanson]

One other note, I'm checking to see if we need a fields/ecs.yml file or not. In the past, we did, otherwise elastic-package would complain about the ECS fields, but that doesn't seem like the case now.

[marioschaefer]

@taylor-swanson could you pls check?

[taylor-swanson]

/test

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[taylor-swanson]

Do we anticipate this list of fields ever to change? Specifically, could any of these fields become optional in the future?

# File: /etc/goflow2/mapping.yaml
formatter:
    fields: # list of fields to format in JSON
        - type
        - time_flow_start_ns
        - sampler_address
        - sequence_num
        - in_if
        - out_if
        - src_addr
        - dst_addr
        - etype
        - proto
        - src_port
        - dst_port
        - src_vlan
        - dst_vlan
        - sampling_rate
        - bytes

If yes, then I would advocate for null checks/ignore_missing on every (applicable) processor. Otherwise I'm fine with leaving those off for now. Since we've specified this in the documentation, we know they should be there and it isn't as critical to add null checks and ignore_missing everywhere.

[taylor-swanson]

/test

[taylor-swanson]

LGTM

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f030595

History

• 💚 Build #14477 succeeded bbc1ab5
• 💔 Build #14301 failed 45bcdbf
• 💔 Build #14012 failed 8dd427c

[elasticmachine]

Package goflow2 - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=goflow2