[Abstraction] Configuration-based teleportation of signals

[timmc-edx]

Currently, each IDA that wants to publish events must add a @receiver-decorated function that listens for a signal and calls the event-bus producer (specifying a topic and key-field as well). It would probably be better to instead have openedx-events do this automatically. Perhaps we could have settings like the following:

EVENT_BUS_AUTOSEND = [
  {
    'signal': 'org.openedx.content_authoring.course.catalog_info.changed.v1',
    'topic': 'course-catalog-info-changed',
    'key_field': 'catalog_info.course_key',
  },
  ...
]

• IDAs could still hardcode the relationship between signals and events when there's a more complicated relationship, but would not be required to for the simple, common case.
• IDAs would have to specify openedx-events as an installed-app or otherwise allow it to perform some initialization at startup (so it could read the config and attach its own signal receivers).
• Depending on the outcome of [Abstraction] Decide how to handle key deserialization when key schema might vary #86 the key-field may not be necessary to specify.

[ormsbee]

Questions (some of them are likely silly, but it's been a while since my head's really been in this space):

1. Can we auto-send everything?
2. Can individual event bus implementations derive the topic from the signal identifier?
3. If (1) is not possible, but (2) is, can the configuration just be a list of the events themselves, like:

EVENT_BUS_AUTOSEND = [
    STUDENT_REGISTRATION_COMPLETED,
    SESSION_LOGIN_COMPLETED,
    # etc.
]

[timmc-edx]

I'd love to hardcode the topic on the OpenEdxPublicSignal.

We could auto-send everything (iterate over list of OpenEdxPublicSignal, attach receiver to each) but then we'd definitely need to work on #79 first. :-)

[ormsbee]

I'd love to hardcode the topic on the OpenEdxPublicSignal.

FWIW, my hope would be that the bus implementation could derive a topic from the signal identifier, e.g. org.openedx.content_authoring.course.catalog_info.changed.v1 gets mapped to content_authoring:course:catalog_info or something.

[timmc-edx]

Ah, I see what you mean. Although if we went that route, would we really want the implementations doing it, or the abstraction layer on top of it (something in openedx-events)?

[ormsbee]

I'm fine with doing that translation at the Kafka driver layer. In the worst case, there's a little extra duplication for different transports, and we can create that separate abstraction later. But I don't think we'll really know what would go in that middle layer until we have two or three of these. For instance, Redis streams doesn't really have partitioning, so it has to overload stream naming for that. That's a kind of mapping decision that's local to Redis's particular capabilities, and I don't think we need to make an abstraction layer for it.

Honestly, I kind of hope that there's no middle abstraction layer at all–that everything can be done with just the in-process stuff in openedx-events, and then a separate kafka-aware library. I'd like to be able to run Kafka and Redis streams concurrently if necessary (for migration purposes), because neither of them know about the other.

[timmc-edx]

@rgraber I think this is completed as of openedx/edx-platform#33642

[timmc-edx]

There are a couple of places where IDAs still call get_producer directly, but I've notified owners of that code and they can make the switch if they'd like.