Wiki article and example YAML/scripts for custom Zeek/Suricata/NetFlow

[philrz]

The motivation for this article is described in #64. Having written the initial version and tested it extensively, I'm inclined to wait on fixes for some of the bugs that were found along the way, as that way the final version will have fewer warts. Beyond the bugs, the end-to-end shown here also illustrates the need for some of the other enhancements waiting in our backlog. I'm starting this out as a draft PR and will add comments at the relevant spots.

Closes #64
Closes #8

[philrz]

I know it's been debated at times whether it's appropriate to use these wide schemas, particularly when it might result in records with excessive nulls because of alerts that don't use several fields. However, I've continued to err on the side of "wide"-ness because we've had community users report it as a bug when data they're accustomed to seeing in columnar layouts in Brim suddenly lose their column headers because additional schemas have appeared. We did discuss once as a group some UX we could add in Brim to call a user's attention to the multiple schemas to make it easy for the user to see the data under each schema separately (e.g. perhaps under a separate "tab" per schema), and when that exists, I'd be more open to letting through more diverse schemas.

[philrz]

Note the histogram is empty, which is unsurprising since I think we're still wired to only populate this if the field named _path is present. We've discussed in the past that it might be helpful to be able to specify a primary "segmentation axis" for the data in a pool (e.g. _path for Zeek) that could be used to make stacked bars when present, and in its absence, just give non-stacked "count" bars? (see brimdata/zui#1541)

[philrz]

The markdown link checker is an expected failure because the links in the docs are relative to where this will ultimately be published in the wiki.

[jameskerr]

As I was reading about the output files and globs, I wasn't sure what directory those files would end up in? The current directory?

Otherwise very clear and high quality. The ergonomics are not quite there in the tooling yet. There are a lot of workarounds.

[philrz]

Thanks for the feedback, @jameskerr. Some thoughts:

As I was reading about the output files and globs, I wasn't sure what directory those files would end up in? The current directory?

I'm glad you spoke up about this. I was on the fence about whether to get into the topic in this article, but since you spotted its absence, I went ahead and added some text that mentions how it goes to a temporary directory by default, and then a new "Debug" section that describes how to use the workdir: option to force it to write to a specific location. Feel free to give it another read and let me know if you think it could use more improvements.

The ergonomics are not quite there in the tooling yet. There are a lot of workarounds.

That's exactly what I was trying to communicate, so thanks for saying that! 😄   Indeed, I'm glad to see that these things are feasible with a little elbow grease. But before I feel confident pointing users at it and implying that we're ready for them to go as far as they want with it, it'll indeed be good to see more bug fixes and UX improvements start to arrive. As mentioned at our team sync meeting, I'm keen to hold this PR open in a draft state and use it like a checklist as we march toward bundling the next release.