[🐛 Bug]: Alerts From Kafka Provider Failed

[Riete]

Describe the bug
if json message which missing "lastReceived" key sent to kafka topic, the alerts will not generated

To Reproduce
Steps to reproduce the behavior:

1. add a kafka provider
2. send some json messages without "lastReceived" key

Expected behavior
can receive alerts

Screenshots
N/A

Additional context
After some research and add trace code, I found when the messages sent to kafka topic missing " lastReceived" key, it will report "Error pushing alert to API", and trace is as following:

{"asctime": "2024-12-13 10:02:06,099", "message": "1 validation error for AlertDto\nlastReceived\n  fromisoformat: argument must be str (type=type_error)", "levelname": "WARNING", "name": "275302b5bf8d48a38e6461d990370706", "filename": "kafka_provider.py", "otelTraceID": "0", "otelSpanID": "0", "otelServiceName": "keep-api", "threadName": "consumer-<keep.providers.kafka_provider.kafka_provider.KafkaProvider object at 0x7faf26897250>", "process": 20, "module": "kafka_provider", "otelTraceSampled": false, "tenant_id": "keep", "workflow_id": null, "workflow_execution_id": null, "context": [{}]}
Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/keep/providers/kafka_provider/kafka_provider.py", line 256, in start_consume
    self._push_alert(record.value)
  File "/venv/lib/python3.11/site-packages/keep/providers/base/base_provider.py", line 681, in _push_alert
    alert_model = AlertDto(
                  ^^^^^^^^^
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for AlertDto
lastReceived
  fromisoformat: argument must be str (type=type_error)

{"asctime": "2024-12-13 10:02:06,100", "message": "Exception occurred", "levelname": "ERROR", "name": "root", "filename": "kafka_provider.py", "otelTraceID": "0", "otelSpanID": "0", "otelServiceName": "keep-api", "threadName": "consumer-<keep.providers.kafka_provider.kafka_provider.KafkaProvider object at 0x7faf26897250>", "process": 20, "module": "kafka_provider", "exc_info": "Traceback (most recent call last):\n  File \"/venv/lib/python3.11/site-packages/keep/providers/kafka_provider/kafka_provider.py\", line 256, in start_consume\n    self._push_alert(record.value)\n  File \"/venv/lib/python3.11/site-packages/keep/providers/base/base_provider.py\", line 681, in _push_alert\n    alert_model = AlertDto(\n                  ^^^^^^^^^\n  File \"pydantic/main.py\", line 341, in pydantic.main.BaseModel.__init__\npydantic.error_wrappers.ValidationError: 1 validation error for AlertDto\nlastReceived\n  fromisoformat: argument must be str (type=type_error)", "otelTraceSampled": false}

if sent messsage contains "lastReceived" key, the alerts will generate normally

[dosubot]

Hey @Riete! I'm here to help with any bugs, questions, or contributions you have. Let's tackle this issue together!

To resolve the issue where alerts are not generated due to a missing "lastReceived" key in a JSON message sent to a Kafka topic, you can modify the code to ensure that the lastReceived field is set to a default value if it is missing. The code should include a mechanism to handle this situation:

# Make sure the lastReceived is a valid date string
# tb: we do this because `AlertDto` object lastReceived is a string and not a datetime object
# TODO: `AlertDto` object `lastReceived` should be a datetime object so we can easily validate with pydantic
if not formatted_event.lastReceived:
    formatted_event.lastReceived = datetime.datetime.now(
        tz=datetime.timezone.utc
    ).isoformat()
else:
    try:
        dateutil.parser.isoparse(formatted_event.lastReceived)
    except ValueError:
        logger.warning("Invalid lastReceived date, setting to now")
        formatted_event.lastReceived = datetime.datetime.now(
            tz=datetime.timezone.utc
        ).isoformat()

This code snippet checks if the lastReceived field is missing or invalid and sets it to the current time in ISO format. Ensure that this logic is correctly integrated into your alert processing pipeline to handle cases where the lastReceived key is missing ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[talboren]

Thanks for opening the issue @Riete! We will fix it right away

[talboren]

@Riete Pushed a quick fix for this one. It should be fine now, if you pull the latest version. Let me know if it still occurs somehow.

[shahargl]

@Riete - @talboren fixed it, you can use the new release 0.31.8 with the fix.