Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instantiate SR only once per sensor - Kafka Trigger Sensor #3380

Open
Thomasr-02 opened this issue Dec 3, 2024 · 2 comments
Open

Instantiate SR only once per sensor - Kafka Trigger Sensor #3380

Thomasr-02 opened this issue Dec 3, 2024 · 2 comments
Labels
bug Something isn't working stale

Comments

@Thomasr-02
Copy link
Contributor

Thomasr-02 commented Dec 3, 2024

Describe the bug

When a sensor is frequently triggered, it results in over X requests per second (rq/s) to the Schema Registry. This level of traffic is unnecessary and causes performance issues in a shared Schema Registry, leading to degraded functionality. While the library used for schema management is capable of caching, the cache is not utilized effectively due to individual triggers for each event.

To Reproduce
Steps to reproduce the behavior:

  1. Create a kafka sensor
  2. Trigger the kafka sensor
  3. Monitoring schema registry requests

Expected behavior
The sensor should instantiate and retrieve the schema from the registry only once per sensor activation. This approach reduces the total number of requests made to the Schema Registry, improving overall performance and efficiency.

Details
The instante SR in kafka.go already doing 1 call and 1 call for get schema, after them seeing it on the riferrei/srclient lib used, cache already implemented by default. The problem here is about many insantiate of trigger called by sensorCtx.triggerActions

Screenshots
image
Requests per second to SR

Environment (please complete the following information):

  • Kubernetes: 1.27.13
  • Argo Events: 1.7.5

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

@Thomasr-02 Thomasr-02 added the bug Something isn't working label Dec 3, 2024
@Thomasr-02 Thomasr-02 changed the title Cache Schema Registry Schema - Kafka Trigger Sensor Cache Schema Registry Client - Kafka Trigger Sensor Dec 7, 2024
@Thomasr-02 Thomasr-02 changed the title Cache Schema Registry Client - Kafka Trigger Sensor Instantiate SR only once per sensor - Kafka Trigger Sensor Dec 10, 2024
@Thomasr-02
Copy link
Contributor Author

Copy link
Contributor

github-actions bot commented Feb 9, 2025

This issue has been automatically marked as stale because it has not had
any activity in the last 60 days. It will be closed if no further activity
occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Feb 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working stale
Projects
None yet
Development

No branches or pull requests

1 participant