feat: add support for annotated python dicts as avro map type #433

mariajgrimaldi · 2024-12-12T18:14:30Z

Description

This PR supports Python dict types mapped to Avro Map type for avro schema generation, aiming to add support to a broader list of events payloads.

This PR also refactors forum-related events so they can be sent through the event bus. Because of backward compatibility, those changes should be studied in a different PR but are here for testing simplicity.

Supporting information

This PR addresses #428 (comment)

Testing instructions

To test with the event bus:

Install this branch into your environment
Install an event bus backend like the redis implementation in the LMS
Create a tutor plugin with this configuration so the event is produced to redis:

from tutor import hooks

redis_config = [
    "EVENT_BUS_PRODUCER = 'edx_event_bus_redis.create_producer'",
    "EVENT_BUS_REDIS_CONNECTION_URL = 'redis://@redis:6379/'",
    "EVENT_BUS_TOPIC_PREFIX = 'dev'",
    "EVENT_BUS_CONSUMER = 'edx_event_bus_redis.RedisEventConsumer'",
]

event_bus_config = """
EVENT_BUS_PRODUCER_CONFIG = {
    'org.openedx.learning.forum.thread.created.v1': {
         'forum-thread-created': {'event_key_field': 'thread.id', 'enabled': True},
     },
     'org.openedx.learning.forum.thread.response.created.v1': {
         'forum-thread-response-created': {'event_key_field': 'thread.id', 'enabled': True},
     },
     'org.openedx.learning.forum.thread.response.comment.created.v1': {
         'forum-thread-response-comment-created': {'event_key_field': 'thread.id', 'enabled': True},
     },
}
"""

hooks.Filters.ENV_PATCHES.add_item(
    (
        "openedx-common-settings", event_bus_config
    )
)

You should see a log similar to this in the LMS:

Deadline

None

Other information

This was previously attempted here: but using record type instead of the map type: #232

Checklists

Check off if complete or not applicable:

Merge Checklist:

Post Merge:

Create a tag
Create a release on GitHub
Check new version is pushed to PyPI after tag-triggered build is
finished.
Delete working branch (if not needed anymore)
Upgrade the package in the Open edX platform requirements (if applicable)

openedx-webhooks · 2024-12-12T18:14:35Z

Thanks for the pull request, @mariajgrimaldi!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently maintained by @openedx/hooks-extension-framework. Tag them in a comment and let them know that your changes are ready for review.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

Ian2012 · 2024-12-12T18:41:19Z

openedx_events/event_bus/avro/tests/test_avro.py

+        dict[str, str]: {'key': 'value'},
+        dict[str, int]: {'key': 1},
+        dict[str, float]: {'key': 1.0},
+        dict[str, bool]: {'key': True},


what about union types:

dict[str, Union[str, int]]: ...,

I added more test cases and this covers even more than I initially thought: a701f78. Thanks for the suggestion!

Ian2012 · 2024-12-13T19:32:59Z

openedx_events/event_bus/avro/schema.py

+        # returns types of dict contents
+        # if data_type == Dict[str, int], arg_data_type = (str, int)


nit: remove code comments

Ian2012 · 2024-12-13T19:34:00Z

openedx_events/event_bus/avro/schema.py

+        if not arg_data_type:
+            raise TypeError(


Can you add a failure test case?

Ian2012 · 2024-12-13T19:34:21Z

openedx_events/event_bus/avro/schema.py

+            )
+        avro_type = SIMPLE_PYTHON_TYPE_TO_AVRO_MAPPING.get(arg_data_type[1])
+        if avro_type is None:
+            raise TypeError(


Ian2012 · 2024-12-13T19:34:31Z

openedx_events/event_bus/avro/serializer.py

@@ -69,6 +69,7 @@ def _event_data_to_avro_record_dict(event_data, serializers=None):
    def value_to_dict(value):
        # Case 1: Value is an instance of an attrs-decorated class
        if hasattr(value, "__attrs_attrs__"):
+            print("\n\n MY VALUE IN VALUE TO DICT", value, "\n\n")


nit: remove unnecessary code

Ian2012 · 2024-12-13T19:36:37Z

openedx_events/learning/signals.py

@@ -297,7 +297,7 @@
 # .. event_data: DiscussionThreadData
 # .. event_warning: This event is currently incompatible with the event bus, list/dict cannot be serialized yet
 FORUM_THREAD_CREATED = OpenEdxPublicSignal(
-    event_type="org.openedx.learning.thread.created.v1",
+    event_type="org.openedx.learning.forum.thread.created.v1",


Does this need to be changed?
Should this emit a FORUM_THREAD_CREATED_V2 signal?

I imagine this will break existing code

Oh yes, as I mentioned in the cover letter those changes are there for testing. That should be addressed in a follow up PR.

Ian2012

Some nits and concerns around changing existing code and some code cleanup

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Dec 12, 2024

mariajgrimaldi mentioned this pull request Dec 12, 2024

docs: [FC-0074] add how-to add event bus support to an Open edX Event #428

Open

Ian2012 reviewed Dec 12, 2024

View reviewed changes

mariajgrimaldi changed the title ~~Mjg/add support for dicts~~ feat: add support for annotated python dicts as avro map type Dec 13, 2024

Ian2012 and others added 7 commits December 13, 2024 17:22

feat: add support for Avro dict

7825196

refactor: add support for dicts as avro maps instead of record

9aaa04e

chore: generate avro schema for each forum event

375a6ab

refactor: rewrite forum payload to be sent to the event bus

1a5c8a8

refactor: regenerate avro schemas for latest forum payload

7f20658

test: add more test cases for serializing dicts

c503783

refactor: address quality issues from latest commits

acdcaa9

mariajgrimaldi force-pushed the MJG/add-support-for-dicts branch from e71efcc to acdcaa9 Compare December 13, 2024 16:23

mariajgrimaldi marked this pull request as ready for review December 13, 2024 18:42

mariajgrimaldi requested a review from a team as a code owner December 13, 2024 18:42

Ian2012 reviewed Dec 13, 2024

View reviewed changes

Ian2012 requested changes Dec 13, 2024

View reviewed changes

mariajgrimaldi requested a review from bmtcril December 13, 2024 19:38

Ian2012 approved these changes Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for annotated python dicts as avro map type #433

feat: add support for annotated python dicts as avro map type #433

mariajgrimaldi commented Dec 12, 2024 •

edited

Loading

openedx-webhooks commented Dec 12, 2024 •

edited

Loading

Ian2012 Dec 12, 2024

mariajgrimaldi Dec 13, 2024

Ian2012 Dec 13, 2024

Ian2012 Dec 13, 2024

Ian2012 Dec 13, 2024

Ian2012 Dec 13, 2024

Ian2012 Dec 13, 2024

mariajgrimaldi Dec 13, 2024

Ian2012 left a comment

		# returns types of dict contents
		# if data_type == Dict[str, int], arg_data_type = (str, int)

feat: add support for annotated python dicts as avro map type #433

Are you sure you want to change the base?

feat: add support for annotated python dicts as avro map type #433

Conversation

mariajgrimaldi commented Dec 12, 2024 • edited Loading

Description

Supporting information

Testing instructions

Deadline

Other information

Checklists

openedx-webhooks commented Dec 12, 2024 • edited Loading

What's next?

🔘 Get product approval

🔘 Provide context

🔘 Get a green build

🔘 Let us know that your PR is ready for review:

Who will review my changes?

Where can I find more information?

When can I expect my changes to be merged?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ian2012 left a comment

Choose a reason for hiding this comment

mariajgrimaldi commented Dec 12, 2024 •

edited

Loading

openedx-webhooks commented Dec 12, 2024 •

edited

Loading