From f1d16d94b2edee4a281f22f5f447bd8a9db3c0eb Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 4 Dec 2024 21:56:20 +0100 Subject: [PATCH 01/13] docs: add how-to add event bus support to an Open edX Event --- .../adding-event-bus-support-to-an-event.rst | 211 ++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 docs/how-tos/adding-event-bus-support-to-an-event.rst diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst new file mode 100644 index 00000000..b9162fd2 --- /dev/null +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -0,0 +1,211 @@ +Adding Event Bus Support to an Open edX Event +============================================= + +Before sending an event across services, you need to ensure that the event is compatible with the Open edX Event Bus. This involves ensuring that the event, with its corresponding payload, can be emitted by a service through the event bus and that it can be consumed by other services. This guide will walk you through the process of adding event bus support to an Open edX event. + +For more details on how the :term:`Event Payload` is structured refer to the :doc:`../decisions/0003-events-payload` decision record. + +.. note:: + This guide assumes that you have already created an Open edX event. If you haven't, refer to the :doc:`../how-tos/creating-new-events` how-to guide. + +Step 1: Does my Event Need Event Bus Support? +---------------------------------------------- + +By default, Open edX Events should be compatible with the Open edX Event Bus. However, there are cases when the support might not be possible or needed for a particular event. Here are some scenarios where you might not need to add event bus support: + +- The event is only used within the same application process and cannot be scoped to other services. +- The :term:`Event Payload` contains data types that are not supported by the event bus, and it is not possible to refactor the :term:`Event Payload` to use supported data types. + +When adding support is not possible do the following: + +- Add it to the ``KNOWN_UNSERIALIZABLE_SIGNALS`` list in the ``openedx_events/tooling.py`` file so the event bus ignores it. +- Add a ``warning`` in the event's docstring to inform developers that the event is not compatible with the event bus. + +If you don't add the event to the ``KNOWN_UNSERIALIZABLE_SIGNALS`` list, the CI/CD pipeline will fail for the missing Avro schema that could not be generated for the :term:`Event Payload`. If you don't add a warning in the event's docstring, developers might try to send the event across services and encounter issues. + +Step 2: Define the Event Payload +-------------------------------- + +An Open edX Event is compatible with the event bus when its payload can be serialized, sent, and deserialized by other services. The payload, structured as attrs data classes, must align with the event bus schema format which in this case is the `Avro`_ schema. This ensures the event can be sent by the producer and be then re-emitted by the same instance of `OpenEdxPublicSignal`_ on the consumer side. For more information on the event bus schema format, refer to the :doc:`../decisions/0005-external-event-schema-format` and :doc:`../decisions/0004-external-event-bus-and-django-signal-events` decision records. + +Here is an example of an :term:`Event Payload` structured as attrs data classes that align with the event bus schema format: + +.. code-block:: python + + @attr.s(frozen=True) + class UserNonPersonalData: + """ + Attributes defined for Open edX user object based on non-PII data. + + Arguments: + id (int): unique identifier for the Django User object. + is_active (bool): indicates whether the user is active. + """ + + id = attr.ib(type=int) + is_active = attr.ib(type=bool) + + @attr.s(frozen=True) + class UserPersonalData: + """ + Attributes defined for Open edX user object based on PII data. + + Arguments: + username (str): username associated with the Open edX user. + email (str): email associated with the Open edX user. + name (str): name associated with the Open edX user's profile. + """ + + username = attr.ib(type=str) + email = attr.ib(type=str) + name = attr.ib(type=str, factory=str) + + @attr.s(frozen=True) + class UserData(UserNonPersonalData): + """ + Attributes defined for Open edX user object. + + This class extends UserNonPersonalData to include PII data completing the + user object. + + Arguments: + pii (UserPersonalData): user's Personal Identifiable Information. + """ + + pii = attr.ib(type=UserPersonalData) + + @attr.s(frozen=True) + class CourseData: + """ + Attributes defined for Open edX Course Overview object. + + Arguments: + course_key (str): identifier of the Course object. + display_name (str): display name associated with the course. + start (datetime): start date for the course. + end (datetime): end date for the course. + """ + + course_key = attr.ib(type=CourseKey) + display_name = attr.ib(type=str, factory=str) + start = attr.ib(type=datetime, default=None) + end = attr.ib(type=datetime, default=None) + +The data types used in the attrs classes that the current Open edX Event Bus with the chosen schema are: + +Primitive Data Types +~~~~~~~~~~~~~~~~~~~~ + +- Boolean +- Integer +- Float +- String +- Bytes + +Complex Data Types +~~~~~~~~~~~~~~~~~~ + +- Type-annotated Lists (e.g., ``List[int]``, ``List[str]``) +- Attrs Classes (e.g., ``UserNonPersonalData``, ``UserPersonalData``, ``UserData``, ``CourseData``) +- Types with Custom Serializers (e.g., ``CourseKey``, ``datetime``) + +Ensure that the :term:`Event Payload` is structured as `attrs data classes`_ and that the data types used in those classes align with the event bus schema format. + +Step 3: Ensure Serialization and Deserialization +------------------------------------------------ + +Before sending the event across services, you need to ensure that the :term:`Event Payload` can be serialized and deserialized correctly. The event bus concrete implementations use the Avro schema to serialize and deserialize the :term:`Event Payload` as mentioned in the :doc:`../decisions/0005-external-event-schema-format` decision record. The concrete implementation of the event bus handles the serialization and deserialization with the help of methods implemented by this library. + +.. For example, here's how the Redis event bus handles serialization before sending a message: + +.. .. code-block:: python +.. :emphasize-lines: 4 + +.. # edx_event_bus_redis/internal/producer.py +.. full_topic = get_full_topic(topic) +.. context.full_topic = full_topic +.. event_bytes = serialize_event_data_to_bytes(event_data, signal) +.. message = RedisMessage(topic=full_topic, event_data=event_bytes, event_metadata=event_metadata) +.. stream_data = message.to_binary_dict() + +.. Where `serialize_event_data_to_bytes`_ is a method that serializes the :term:`Event Payload` to bytes using the Avro schema. While the consumer side deserializes the :term:`Event Payload` using the Avro schema with the help of the `deserialize_bytes_to_event_data`_ method: + +.. .. code-block:: python +.. :emphasize-lines: 3 + +.. # edx_event_bus_redis/internal/consumer.py +.. signal = OpenEdxPublicSignal.get_signal_by_type(msg.event_metadata.event_type) +.. event_data = deserialize_bytes_to_event_data(msg.event_data, signal) +.. send_results = signal.send_event_with_custom_metadata(msg.event_metadata, **event_data) + +If the :term:`Event Payload` contains types that are not supported by the event bus, you could implement custom serializers for these types. This ensures that the :term:`Event Payload` can be serialized and deserialized correctly when sent across services. + +Here is an example of a custom serializer for the ``CourseKey`` type: + +.. code-block:: python + + # event_bus/avro/custom_serializers.py + class CourseKeyAvroSerializer(BaseCustomTypeAvroSerializer): + """ + CustomTypeAvroSerializer for CourseKey class. + """ + + cls = CourseKey + field_type = PYTHON_TYPE_TO_AVRO_MAPPING[str] + + @staticmethod + def serialize(obj) -> str: + """Serialize obj into string.""" + return str(obj) + + @staticmethod + def deserialize(data: str): + """Deserialize string into obj.""" + return CourseKey.from_string(data) + + +After implementing the serializer, add it to ``DEFAULT_CUSTOM_SERIALIZERS`` at the end of the ``event_bus/avro/custom_serializers.py`` file: + +.. code-block:: python + + DEFAULT_CUSTOM_SERIALIZERS = [ + # Other custom serializers + CourseKey: CourseKeyAvroSerializer, + ] + +Now the :term:`Event Payload` can be serialized and deserialized correctly when sent across services. + +.. warning:: + One of the known limitations of the current Open edX Event Bus is that it does not support dictionaries as data types. If the :term:`Event Payload` contains dictionaries, you may need to refactor the :term:`Event Payload` to use supported data types. When you know the structure of the dictionary, you can create an attrs class that represents the dictionary structure. If not, you can use a str type to represent the dictionary as a string and deserialize it on the consumer side using JSON deserialization. + +If your :term:`Event Payload` contains only supported data types, you can skip this step. + +Step 4: Generate the Avro Schema +-------------------------------- + +As mentioned in the previous step, the serialization and deserialization of the :term:`Event Payload` is handled by the concrete event bus implementation with the help of methods implemented in this library. However, ``openedx-events`` ensures the payload of new events can be serialized and deserialized correctly by adding checks in the CI/CD pipeline for schema verification. To ensure this, you need to generate the Avro schema for the :term:`Event Payload`: + +1. Run the following command to generate the Avro schema for the :term:`Event Payload`: + +.. code-block:: bash + + python manage.py generate_avro_schemas org.openedx.learning.course.enrollment.changed.v1 + +2. The Avro schema for the :term:`Event Payload` will be generated in the ``openedx_events/event_bus/avro/tests/schemas`` directory. +3. Push the changes to the branch and create a pull request or run the checks locally to verify that the Avro schema was generated correctly. + +.. code-block:: bash + + make test + +Step 5: Send the Event Across Services with the Event Bus +--------------------------------------------------------- + +To validate that you can consume the event emitted by a service through the event bus, you can send the event across services. Here is an example of how you can send the event across services using the Redis event bus implementation following the `setup instructions in a Tutor environment`_. + +.. _Avro: https://avro.apache.org/ +.. _OpenEdxPublicSignal: https://github.com/openedx/openedx-events/blob/main/openedx_events/tooling.py#L37 +.. _attrs data classes: https://www.attrs.org/en/stable/overview.html +.. _serialize_event_data_to_bytes: https://github.com/openedx/openedx-events/blob/main/openedx_events/event_bus/avro/serializer.py#L82-L98 +.. _deserialize_bytes_to_event_data: https://github.com/openedx/openedx-events/blob/main/openedx_events/event_bus/avro/deserializer.py#L86-L98 +.. _setup instructions in a Tutor environment: https://github.com/openedx/event-bus-redis/blob/main/docs/tutor_installation.rst From 5d435b0fbfafa5bf91d25031ce639cf16ae26534 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 5 Dec 2024 20:44:55 +0100 Subject: [PATCH 02/13] docs: link doc about adding event bus support to an event --- docs/how-tos/using-the-event-bus.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/how-tos/using-the-event-bus.rst b/docs/how-tos/using-the-event-bus.rst index 17a9256d..69d79c15 100644 --- a/docs/how-tos/using-the-event-bus.rst +++ b/docs/how-tos/using-the-event-bus.rst @@ -1,6 +1,8 @@ Using the Open edX Event Bus ============================ +.. note:: Be sure to check out how to make your Open edX Event event bus compatible in the :doc:`../how-tos/adding-event-bus-support-to-an-event` guide. + After creating a new Open edX Event, you might need to send it across services instead of just within the same process. For this kind of use-cases, you might want to use the Open edX Event Bus. Here :doc:`../concepts/event-bus`, you can find useful information to start getting familiar with the Open edX Event Bus. The Open edX Event Bus is a key component of the Open edX architecture, enabling services to communicate without direct dependencies on each other. This guide provides an overview of how to use the event bus to broadcast Open edX Events to multiple services, allowing them to react to changes or actions in the system. From b814865cddd4541935281d2783c73ae1e01dfae5 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 5 Dec 2024 20:49:32 +0100 Subject: [PATCH 03/13] fix: use correct reference removed after rebase --- docs/how-tos/index.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst index 45b871bf..5a85d3c1 100644 --- a/docs/how-tos/index.rst +++ b/docs/how-tos/index.rst @@ -7,6 +7,8 @@ How-tos creating-new-events adding-events-to-a-service + adding-event-bus-support-to-an-event + using-the-event-bus using-events using-the-event-bus add-new-event-bus-concrete-implementation From 457edf9781df7690ac5291eeda09d0dfbb500a3b Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 5 Dec 2024 21:09:41 +0100 Subject: [PATCH 04/13] docs: add glossary references for avro schema --- docs/how-tos/adding-event-bus-support-to-an-event.rst | 10 ++++++---- docs/reference/glossary.rst | 7 +++++++ 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index b9162fd2..5fc543f5 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -26,9 +26,11 @@ If you don't add the event to the ``KNOWN_UNSERIALIZABLE_SIGNALS`` list, the CI/ Step 2: Define the Event Payload -------------------------------- -An Open edX Event is compatible with the event bus when its payload can be serialized, sent, and deserialized by other services. The payload, structured as attrs data classes, must align with the event bus schema format which in this case is the `Avro`_ schema. This ensures the event can be sent by the producer and be then re-emitted by the same instance of `OpenEdxPublicSignal`_ on the consumer side. For more information on the event bus schema format, refer to the :doc:`../decisions/0005-external-event-schema-format` and :doc:`../decisions/0004-external-event-bus-and-django-signal-events` decision records. +An Open edX Event is compatible with the event bus when its payload can be serialized, sent, and deserialized by other services. The payload, structured as `attrs data classes`_, must align with the event bus schema format which in this case is the :term:`Avro Schema`. This schema is used to serialize and deserialize the :term:`Event Payload` when sending it across services. -Here is an example of an :term:`Event Payload` structured as attrs data classes that align with the event bus schema format: +This ensures the event can be sent by the producer and be then re-emitted by the same instance of `OpenEdxPublicSignal`_ on the consumer side. For more information on the event bus schema format, refer to the :doc:`../decisions/0004-external-event-bus-and-django-signal-events` and :doc:`../decisions/0005-external-event-schema-format` decision records. + +Here is an example of an :term:`Event Payload` structured as `attrs data classes`_ that align with the event bus schema format: .. code-block:: python @@ -114,7 +116,7 @@ Ensure that the :term:`Event Payload` is structured as `attrs data classes`_ and Step 3: Ensure Serialization and Deserialization ------------------------------------------------ -Before sending the event across services, you need to ensure that the :term:`Event Payload` can be serialized and deserialized correctly. The event bus concrete implementations use the Avro schema to serialize and deserialize the :term:`Event Payload` as mentioned in the :doc:`../decisions/0005-external-event-schema-format` decision record. The concrete implementation of the event bus handles the serialization and deserialization with the help of methods implemented by this library. +Before sending the event across services, you need to ensure that the :term:`Event Payload` can be serialized and deserialized correctly. The event bus concrete implementations use the :term:`Avro Schema` to serialize and deserialize the :term:`Event Payload` as mentioned in the :doc:`../decisions/0005-external-event-schema-format` decision record. The concrete implementation of the event bus handles the serialization and deserialization with the help of methods implemented by this library. .. For example, here's how the Redis event bus handles serialization before sending a message: @@ -183,7 +185,7 @@ If your :term:`Event Payload` contains only supported data types, you can skip t Step 4: Generate the Avro Schema -------------------------------- -As mentioned in the previous step, the serialization and deserialization of the :term:`Event Payload` is handled by the concrete event bus implementation with the help of methods implemented in this library. However, ``openedx-events`` ensures the payload of new events can be serialized and deserialized correctly by adding checks in the CI/CD pipeline for schema verification. To ensure this, you need to generate the Avro schema for the :term:`Event Payload`: +As mentioned in the previous step, the serialization and deserialization of the :term:`Event Payload` is handled by the concrete event bus implementation with the help of methods implemented in this library. However, although openedx-events does not handles the serialization and deserialization of the :term:`Event Payload` directly, it ensures the payload of new events can be serialized and deserialized correctly by adding checks in the CI/CD pipeline for schema verification. To ensure this, you need to generate the Avro schema for the :term:`Event Payload`: 1. Run the following command to generate the Avro schema for the :term:`Event Payload`: diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 6c118c19..81cc8d41 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -41,4 +41,11 @@ An event has multiple components that are used to define, trigger, and handle th Topic How the event bus implementation groups related events, such as streams in Redis. Producers publish events to topics, and consumers subscribe to topics to receive events. + Avro Schema + A specification describing the expected field names and types in an Avro record dictionary. See `Apache Avro`_ for more information. + + Avro Record Dictionary + A dictionary whose structure is determined by an Avro schema. These dictionaries are the entities that are actually serialized to bytes and sent over the wire to the event bus. + .. _Events Payload ADR: :doc: `/decisions/0003-events-payload` +.. _Apache Avro: https://avro.apache.org/docs/current/spec.html From 4f83a8c7c66a47a146f0fa9f79e2eaa9e6c5285c Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 9 Dec 2024 12:35:57 +0100 Subject: [PATCH 05/13] docs: ask for explanation when adding an event not event-bus compatible --- docs/how-tos/adding-event-bus-support-to-an-event.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index 5fc543f5..deea7e95 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -19,7 +19,7 @@ By default, Open edX Events should be compatible with the Open edX Event Bus. Ho When adding support is not possible do the following: - Add it to the ``KNOWN_UNSERIALIZABLE_SIGNALS`` list in the ``openedx_events/tooling.py`` file so the event bus ignores it. -- Add a ``warning`` in the event's docstring to inform developers that the event is not compatible with the event bus. +- Add a ``warning`` in the event's docstring to inform developers that the event is not compatible with the event bus and why. If you don't add the event to the ``KNOWN_UNSERIALIZABLE_SIGNALS`` list, the CI/CD pipeline will fail for the missing Avro schema that could not be generated for the :term:`Event Payload`. If you don't add a warning in the event's docstring, developers might try to send the event across services and encounter issues. From 59a8efa71b35f6a7e95e34f0c5f7a4ee4157fa22 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 9 Dec 2024 12:38:58 +0100 Subject: [PATCH 06/13] docs: drop examples and clarify that data sent/received is identical --- .../adding-event-bus-support-to-an-event.rst | 67 +------------------ 1 file changed, 3 insertions(+), 64 deletions(-) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index deea7e95..90650d8f 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -28,70 +28,7 @@ Step 2: Define the Event Payload An Open edX Event is compatible with the event bus when its payload can be serialized, sent, and deserialized by other services. The payload, structured as `attrs data classes`_, must align with the event bus schema format which in this case is the :term:`Avro Schema`. This schema is used to serialize and deserialize the :term:`Event Payload` when sending it across services. -This ensures the event can be sent by the producer and be then re-emitted by the same instance of `OpenEdxPublicSignal`_ on the consumer side. For more information on the event bus schema format, refer to the :doc:`../decisions/0004-external-event-bus-and-django-signal-events` and :doc:`../decisions/0005-external-event-schema-format` decision records. - -Here is an example of an :term:`Event Payload` structured as `attrs data classes`_ that align with the event bus schema format: - -.. code-block:: python - - @attr.s(frozen=True) - class UserNonPersonalData: - """ - Attributes defined for Open edX user object based on non-PII data. - - Arguments: - id (int): unique identifier for the Django User object. - is_active (bool): indicates whether the user is active. - """ - - id = attr.ib(type=int) - is_active = attr.ib(type=bool) - - @attr.s(frozen=True) - class UserPersonalData: - """ - Attributes defined for Open edX user object based on PII data. - - Arguments: - username (str): username associated with the Open edX user. - email (str): email associated with the Open edX user. - name (str): name associated with the Open edX user's profile. - """ - - username = attr.ib(type=str) - email = attr.ib(type=str) - name = attr.ib(type=str, factory=str) - - @attr.s(frozen=True) - class UserData(UserNonPersonalData): - """ - Attributes defined for Open edX user object. - - This class extends UserNonPersonalData to include PII data completing the - user object. - - Arguments: - pii (UserPersonalData): user's Personal Identifiable Information. - """ - - pii = attr.ib(type=UserPersonalData) - - @attr.s(frozen=True) - class CourseData: - """ - Attributes defined for Open edX Course Overview object. - - Arguments: - course_key (str): identifier of the Course object. - display_name (str): display name associated with the course. - start (datetime): start date for the course. - end (datetime): end date for the course. - """ - - course_key = attr.ib(type=CourseKey) - display_name = attr.ib(type=str, factory=str) - start = attr.ib(type=datetime, default=None) - end = attr.ib(type=datetime, default=None) +This ensures the event can be sent by the producer and be then re-emitted by the same instance of `OpenEdxPublicSignal`_ on the consumer side, guaranteeing that the data sent and received is the identical. Serializing this way should prevent data inconsistencies between services, e.g., timezone issues and precision loss. For more information on the event bus schema format, refer to the :doc:`../decisions/0004-external-event-bus-and-django-signal-events` and :doc:`../decisions/0005-external-event-schema-format` decision records. The data types used in the attrs classes that the current Open edX Event Bus with the chosen schema are: @@ -113,6 +50,8 @@ Complex Data Types Ensure that the :term:`Event Payload` is structured as `attrs data classes`_ and that the data types used in those classes align with the event bus schema format. +In the ``data.py`` files within each architectural subdomain you can find examples of the :term:`Event Payload` structured as `attrs data classes`_ that align with the event bus schema format. + Step 3: Ensure Serialization and Deserialization ------------------------------------------------ From 3dd13993ec1c87ae47b55800c3a7d84ecc75d856 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 9 Dec 2024 12:54:25 +0100 Subject: [PATCH 07/13] docs: rewrite introduction to step-by-step avro schema generation guide --- docs/how-tos/adding-event-bus-support-to-an-event.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index 90650d8f..d1b4b438 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -124,7 +124,7 @@ If your :term:`Event Payload` contains only supported data types, you can skip t Step 4: Generate the Avro Schema -------------------------------- -As mentioned in the previous step, the serialization and deserialization of the :term:`Event Payload` is handled by the concrete event bus implementation with the help of methods implemented in this library. However, although openedx-events does not handles the serialization and deserialization of the :term:`Event Payload` directly, it ensures the payload of new events can be serialized and deserialized correctly by adding checks in the CI/CD pipeline for schema verification. To ensure this, you need to generate the Avro schema for the :term:`Event Payload`: +As mentioned in the previous step, the serialization and deserialization of the :term:`Event Payload` is handled by the concrete event bus implementation with the help of methods implemented in this library. However, although openedx-events does not handles the serialization and deserialization of the :term:`Event Payload` directly, it ensures the payload of new events can be serialized and deserialized correctly by adding checks in the CI/CD pipeline for schema verification. To ensure tests pass, you need to generate an Avro test schema for your new event's :term:`Event Payload`: 1. Run the following command to generate the Avro schema for the :term:`Event Payload`: From a117543fc961326e73871f5fb0fcc7fc61217c0f Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 9 Dec 2024 13:08:23 +0100 Subject: [PATCH 08/13] docs: drop suggestion about skipping section --- docs/how-tos/adding-event-bus-support-to-an-event.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index d1b4b438..a636baee 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -119,8 +119,6 @@ Now the :term:`Event Payload` can be serialized and deserialized correctly when .. warning:: One of the known limitations of the current Open edX Event Bus is that it does not support dictionaries as data types. If the :term:`Event Payload` contains dictionaries, you may need to refactor the :term:`Event Payload` to use supported data types. When you know the structure of the dictionary, you can create an attrs class that represents the dictionary structure. If not, you can use a str type to represent the dictionary as a string and deserialize it on the consumer side using JSON deserialization. -If your :term:`Event Payload` contains only supported data types, you can skip this step. - Step 4: Generate the Avro Schema -------------------------------- From f91291c62efb3a5b946651af7234f63f738598c8 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 9 Dec 2024 13:18:15 +0100 Subject: [PATCH 09/13] docs: use placeholder instead of specific event for avro schema command --- docs/how-tos/adding-event-bus-support-to-an-event.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index a636baee..3b399ae5 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -128,7 +128,9 @@ As mentioned in the previous step, the serialization and deserialization of the .. code-block:: bash - python manage.py generate_avro_schemas org.openedx.learning.course.enrollment.changed.v1 + python manage.py generate_avro_schemas YOUR_EVENT_TYPE + +Run ``python manage.py generate_avro_schemas --help`` to see the available options for the command. 2. The Avro schema for the :term:`Event Payload` will be generated in the ``openedx_events/event_bus/avro/tests/schemas`` directory. 3. Push the changes to the branch and create a pull request or run the checks locally to verify that the Avro schema was generated correctly. From 375c60987c95133e05a95448b7b9652ee4259c5a Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 9 Dec 2024 13:23:15 +0100 Subject: [PATCH 10/13] docs: add note about custom serializer in step 5 --- docs/how-tos/adding-event-bus-support-to-an-event.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index 3b399ae5..acb9a88b 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -144,6 +144,8 @@ Step 5: Send the Event Across Services with the Event Bus To validate that you can consume the event emitted by a service through the event bus, you can send the event across services. Here is an example of how you can send the event across services using the Redis event bus implementation following the `setup instructions in a Tutor environment`_. +.. note:: If you implemented a custom serializer for a type in the :term:`Event Payload`, the custom serializer support must be included in both the producer and consumer sides before it can be used. + .. _Avro: https://avro.apache.org/ .. _OpenEdxPublicSignal: https://github.com/openedx/openedx-events/blob/main/openedx_events/tooling.py#L37 .. _attrs data classes: https://www.attrs.org/en/stable/overview.html From bfc647ccfb6e06b88e12b9eff37e2159cfbd273f Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 11 Dec 2024 13:59:35 +0100 Subject: [PATCH 11/13] docs: add note about reading next event bus how to --- docs/how-tos/adding-event-bus-support-to-an-event.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index acb9a88b..8b84dd08 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -142,7 +142,7 @@ Run ``python manage.py generate_avro_schemas --help`` to see the available optio Step 5: Send the Event Across Services with the Event Bus --------------------------------------------------------- -To validate that you can consume the event emitted by a service through the event bus, you can send the event across services. Here is an example of how you can send the event across services using the Redis event bus implementation following the `setup instructions in a Tutor environment`_. +To validate that you can consume the event emitted by a service through the event bus, you can send the event across services. Here is an example of how you can send the event across services using the Redis event bus implementation following the `setup instructions in a Tutor environment`_. We recommend also following :doc:`../how-tos/using-the-event-bus` to understand how to use the event bus in your environment. .. note:: If you implemented a custom serializer for a type in the :term:`Event Payload`, the custom serializer support must be included in both the producer and consumer sides before it can be used. From a05eb03d4e83a73cf7fd935a7d711c76bc5a8fbd Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Wed, 11 Dec 2024 19:36:30 +0100 Subject: [PATCH 12/13] fix: move reference to right place --- docs/how-tos/index.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst index 5a85d3c1..edb107f7 100644 --- a/docs/how-tos/index.rst +++ b/docs/how-tos/index.rst @@ -7,8 +7,7 @@ How-tos creating-new-events adding-events-to-a-service - adding-event-bus-support-to-an-event - using-the-event-bus using-events + adding-event-bus-support-to-an-event using-the-event-bus add-new-event-bus-concrete-implementation From 3b35a9d1b0addcdcb195cf14de95e44be919e1f6 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 26 Dec 2024 10:56:45 +0100 Subject: [PATCH 13/13] docs: address PR reviews --- .../adding-event-bus-support-to-an-event.rst | 28 ++++--------------- 1 file changed, 6 insertions(+), 22 deletions(-) diff --git a/docs/how-tos/adding-event-bus-support-to-an-event.rst b/docs/how-tos/adding-event-bus-support-to-an-event.rst index 8b84dd08..9138c699 100644 --- a/docs/how-tos/adding-event-bus-support-to-an-event.rst +++ b/docs/how-tos/adding-event-bus-support-to-an-event.rst @@ -14,7 +14,7 @@ Step 1: Does my Event Need Event Bus Support? By default, Open edX Events should be compatible with the Open edX Event Bus. However, there are cases when the support might not be possible or needed for a particular event. Here are some scenarios where you might not need to add event bus support: - The event is only used within the same application process and cannot be scoped to other services. -- The :term:`Event Payload` contains data types that are not supported by the event bus, and it is not possible to refactor the :term:`Event Payload` to use supported data types. +- The :term:`Event Payload` contains data types that are not supported by the event bus, e.g., lists of dictionaries or data attrs classes with unsupported data types; when it is not possible to refactor the :term:`Event Payload` to use supported data types. When adding support is not possible do the following: @@ -23,6 +23,8 @@ When adding support is not possible do the following: If you don't add the event to the ``KNOWN_UNSERIALIZABLE_SIGNALS`` list, the CI/CD pipeline will fail for the missing Avro schema that could not be generated for the :term:`Event Payload`. If you don't add a warning in the event's docstring, developers might try to send the event across services and encounter issues. +.. warning:: Maintainers will check event bus compatibility for new events. To avoid issues, make sure to consider compatibility during the design phase. Contact maintainers if you are unsure about the compatibility of an event. + Step 2: Define the Event Payload -------------------------------- @@ -57,27 +59,7 @@ Step 3: Ensure Serialization and Deserialization Before sending the event across services, you need to ensure that the :term:`Event Payload` can be serialized and deserialized correctly. The event bus concrete implementations use the :term:`Avro Schema` to serialize and deserialize the :term:`Event Payload` as mentioned in the :doc:`../decisions/0005-external-event-schema-format` decision record. The concrete implementation of the event bus handles the serialization and deserialization with the help of methods implemented by this library. -.. For example, here's how the Redis event bus handles serialization before sending a message: - -.. .. code-block:: python -.. :emphasize-lines: 4 - -.. # edx_event_bus_redis/internal/producer.py -.. full_topic = get_full_topic(topic) -.. context.full_topic = full_topic -.. event_bytes = serialize_event_data_to_bytes(event_data, signal) -.. message = RedisMessage(topic=full_topic, event_data=event_bytes, event_metadata=event_metadata) -.. stream_data = message.to_binary_dict() - -.. Where `serialize_event_data_to_bytes`_ is a method that serializes the :term:`Event Payload` to bytes using the Avro schema. While the consumer side deserializes the :term:`Event Payload` using the Avro schema with the help of the `deserialize_bytes_to_event_data`_ method: - -.. .. code-block:: python -.. :emphasize-lines: 3 - -.. # edx_event_bus_redis/internal/consumer.py -.. signal = OpenEdxPublicSignal.get_signal_by_type(msg.event_metadata.event_type) -.. event_data = deserialize_bytes_to_event_data(msg.event_data, signal) -.. send_results = signal.send_event_with_custom_metadata(msg.event_metadata, **event_data) +If you are interested in how the serialization and deserialization of the :term:`Event Payload` is handled by the event bus, you can refer to the concrete event bus implementation in the Open edX Event Bus repository. For example, here's how the Redis event bus handles `serialization`_ and `deserialization`_ when sending and receiving events. If the :term:`Event Payload` contains types that are not supported by the event bus, you could implement custom serializers for these types. This ensures that the :term:`Event Payload` can be serialized and deserialized correctly when sent across services. @@ -152,3 +134,5 @@ To validate that you can consume the event emitted by a service through the even .. _serialize_event_data_to_bytes: https://github.com/openedx/openedx-events/blob/main/openedx_events/event_bus/avro/serializer.py#L82-L98 .. _deserialize_bytes_to_event_data: https://github.com/openedx/openedx-events/blob/main/openedx_events/event_bus/avro/deserializer.py#L86-L98 .. _setup instructions in a Tutor environment: https://github.com/openedx/event-bus-redis/blob/main/docs/tutor_installation.rst +.. _serialization: https://github.com/openedx/event-bus-redis/blob/main/edx_event_bus_redis/internal/producer.py#L128-L137 +.. _deserialization: https://github.com/openedx/event-bus-redis/blob/main/edx_event_bus_redis/internal/consumer.py#L276-L289