Roadmap for Zeek 5.0

[rsmmr]

This is a meta ticket tracking next steps.

For Zeek 5.0:

• Bring back OpenSSL in Implement new ALM transport #201.
• Fix shutdown glitches in Implement new ALM transport #201
• Merge Implement new ALM transport #201 (w/ reduced ALM functionality)
• Fix UBSan "vptr" check failures (Fix UBSan "vptr" check failures zeek#1176)
• Add Websocket-based client API for event submission/subscription, using new serialization (Add new WebSocket interface to Broker #219)
• Check whether we can disable any warnings from CAF. That one is probably tricky, because we are building CAF as part of Broker. Hence, only suppressing warnings from CAF headers wouldn't be sufficient. This means we probably need to add a 'no-warnings' flag to CAF's CMake.
• Ensure Broker protocol will remain backwards compatible with future changes (e.g., serialization format switch)
• Clean up and triage existing tickets

After 5.0:

• Switch Broker protocol to new serialization
• Revisit need for Python bindings
• Revisit full ALM
• Clean up and modernize code where possible
• Decide on new serialization format

[timwoj]

You might add zeek/zeek#1743 here too.

[Neverlord]

Yep, thanks for the pointer. That one fell off the radar as it seems. I'm not sure if there's anything we can do about it, as the stack trace indicates it's crashing in std::random_device. But I'll evaluate it after the ALM merge is done.

[timwoj]

Good point. It could be related to this: rr-debugger/rr#2459, which appears to have been fixed a while ago. I'll comment on the zeek issue about it.

[Neverlord]

Putting this as a reminder here, also probably for after 5.0 since it's just a 'nice-to-have':

• Provide a JSON schema for the new WebSocket protocol.

[Neverlord]

Revisit need for Python bindings.

With WebSockets available in 5.0 as language-agnostic way to communicate to Broker: why not deprecate the bindings with 5.0?

We currently have 6 tickets open that are related to the Python bindings. I think simply closing all of them as 'wontfix' is probably the best way to move forward. If we do want to make interfacing with Broker easier from Python, we could instead provide a couple of (pure Python) utility functions that streamline operating on the WebSocket messages by converting between the Broker JSON format and proper Python types.

[rsmmr]

Yeah, I like that, and had indeed also already thought about providing a new Python module that would help with the WebSocket/JSON encoding.

However, I'd wait one version before deprecating the Python bindings: we're just adding WebSocket now with close-to-zero experience using it; let's give it a cycle before we recommend people move over to it for existing Python clients. That also allows the Python bindings to remain not-deprecated for the 5.0 LTS cycle, giving people plenty time.

So, I propose we do this with 5.1. I agree, though, that we then don't need to work on all those tickets in the meantime; we can just go into maintainance mode.

[Neverlord]

That also allows the Python bindings to remain not-deprecated for the 5.0 LTS cycle, giving people plenty time.

Just for my understanding: what difference does it make to deprecate the bindings in 5.1 versus deprecating with 6.0? In both cases, we'll have to support the Python bindings until 7.0, or does the Zeek policy work differently?

[Neverlord]

Check whether we can disable any warnings from CAF. That one is probably tricky, because we are building CAF as part of Broker. Hence, only suppressing warnings from CAF headers wouldn't be sufficient. This means we probably need to add a 'no-warnings' flag to CAF's CMake.

Zeek and Broker modify global CMake variables that are meant to capture user-configurations, e.g., CMAKE_CXX_FLAGS. Since CAF is integrated as sub-directory, it gets all the warning (and other) flags passed down automatically. Ideally, Zeek and Broker would leave the global flags alone and work with target properties instead. In the current setup, I don't see a sane way to 'drop' the flags that Zeek and Broker have added without ignoring all user-defined flags as well (which we shouldn't do IMHO).

[timwoj]

Ideally, Zeek and Broker would leave the global flags alone and work with target properties instead.

After my fight with c-ares, I completely agree with this. It's something I'd definitely like to fix about our CMake setup.

[rsmmr]

That also allows the Python bindings to remain not-deprecated for the 5.0 LTS cycle, giving people plenty time.

Just for my understanding: what difference does it make to deprecate the bindings in 5.1 versus deprecating with 6.0? In both cases, we'll have to support the Python bindings until 7.0, or does the Zeek policy work differently?

That's correct. My point was that if we had deprecated them in 5.0 already, people would start getting warnings immediately from now on.

[Neverlord]

Is there still something left todo here or can we close this ticket? We have a separate ticket for the Python binding removal / deprecation and the CMake overhaul is underway. 🙂

[timwoj]

Nope, I'm fine with closing it.