Parallel request to bulk service failed with a few jobs in IN_PROGRESS STATE

[chandrams]

Describe the bug
Observed the below issues with parallel requests to bulk service:

• A few jobs are stuck in IN_PROGRESS STATE
• A few jobs had unprocessed experiments with status as completed and with notifications as null

How to reproduce it

• Configure thanos datasource with a single experiment and 15 days of usage results
• Deploy kruize
• Invoke Bulk service without any filters with datasource as thanos

Expected behavior

• Parallel requests to bulk service should complete or fail and not remain pending in IN_PROGRESS state
• Job status reports as completed with unprocessed requirements, should the job status be failed?

Relevant logs

{"status":"IN_PROGRESS","total_experiments":1,"processed_experiments":1,"message":null,"job_id":"f918009a-2e3d-486a-ad22-aa9204ae8c0b","job_start_time":"2024-11-13T08:19:17.990Z","job_end_time":null}

Unprocessed experiments

{
    "status": "COMPLETED",
    "total_experiments": 1,
    "processed_experiments": 1,
    "notifications": null,
    "experiments": {
        "thanos|default|msc-0|tfb-qrh-sample-0(deployment)|tfb-0": {
            "recommendations": {
                "status": "UNPROCESSED",
                "notifications": null
            }
        }
    },

Environment:

• Kubernetes Cluster : openshift

Additional context
Add any other context about the problem here, any links or screenshots