Dataset Request API Not working for New ML PII Data Product in SB Lern BB

[Shakthieshwari]

Hi @anandp504 @manjudr ,

As part of 5.3 Lern Release, We are creating the custom ML PII Data Product in SB Lern BB.
Story Link :- https://project-sunbird.atlassian.net/browse/LR-491
Discussion Forum :- https://github.com/orgs/Sunbird-Lern/discussions/58

Implementation is completed and we started with testing, We found an issue in the below API

1. Proxy Exhaust API - /api/course/v1/jobrequest/submit after performing encryption logic it is internally calling SB Obsrv dataset/v1/request/submit in Dev Lern

curl --location --request POST 'https://dev.lern.sunbird.org/api/course/v1/jobrequest/submit' \ --header 'x-authenticated-user-token: {{access_token}}' \ --header 'Authorization: Bearer {{jwt_token}}' \ --header 'Content-Type: application/json' \ --header 'x-channel-id: 01269878797503692810' \ --header 'x-authenticated-userid: 11f591e6-fea9-4ae1-a787-aea295c7ab8a' \ --data-raw '{ "request": { "tag": "program_6194f8d4db43ec4400a33b7c:01250894314817129555", "requestedBy": "fbe926ac-a395-40e4-a65b-9b4f711d7642", "dataset": "program-user-exhaust", "datasetConfig": { "type": "program-user-exhaust", "params": { "filters": [ { "table_name": "program_enrollment", "table_filters": [ { "name": "program_id", "operator": "=", "value": "6194f8d4db43ec4400a33b7c" } ] }, { "table_name": "user_consent", "table_filters": [ { "name": "object_id", "operator": "=", "value": "6194f8d4db43ec4400a33b7c" }, { "name": "user_id", "operator": "=", "value": "13b24110-af97-430a-9d2c-0dd7ef6dccaa" } ] } ] }, "title": "User Detail Report" }, "output_format": "csv", "encryptionKey": "test1234" } }'

It is throwing an error, 500 Internal Server Error

{
    "id": "api.jobrequest.submit",
    "ver": "v1",
    "ts": "2023-06-01 13:54:09:568+0000",
    "params": {
        "resmsgid": null,
        "msgid": "8c3b002e-8d4e-4449-a9b0-1dccd0b254bb",
        "err": null,
        "status": "SERVER_ERROR",
        "errmsg": null
    },
    "responseCode": "SERVER_ERROR",
    "result": {}
}

We also tried, executing SB Obsrv Dataset API :-

curl --location --request POST 'https://dev.lern.sunbird.org/api/dataset/v1/request/submit' \ --header 'x-authenticated-user-token: {{access_token}}' \ --header 'Authorization: Bearer {{jwt_token}}' \ --header 'Content-Type: application/json' \ --header 'x-channel-id: 01269878797503692810' \ --header 'x-authenticated-userid: 7bc649e4-838f-48ca-948f-e7a0d7d67baa' \ --data-raw '{ "request": { "tag": "602512d8e6aefa27d9629bc3", "requestedBy": "7bc649e4-838f-48ca-948f-e7a0d7d67baa", "dataset": "program-user-exhaust", "datasetConfig": { "type":"program-user-exhaust", "params":{ "filters":[{ "table_name":"program_enrollment", "table_filters":[{ "name":"program_id", "operator":"=", "value":"602512d8e6aefa27d9629bc3" }] }, { "table_name":"user_consent", "table_filters":[{ "name":"object_id", "operator":"=", "value":"602512d8e6aefa27d9629bc3" }] }] }, "title":"User Detail Report" }, "output_format": "csv", "encryptionKey": "test1234" } }'

It also Failed and throwed below error
403 Forbidden- You do not have permission to perform this operation

We just did trail & error method and figured out that, In the Request Body "dataset": "progress-exhaust" If we pass dataset value as "progress-exhaust" API is working fine, but if we pass dataset value as "program-user-exhaust" it is not working. This is a New ML PII Data Product which we are developing as part of 5.3 release, Please let me know if this new data product job_id = "program-user-exhaust" have to be configured somewhere in the dev obsrv for the API to work

Below is the Curl which was success by passing "dataset": "progress-exhaust"
curl --location --request POST 'https://dev.lern.sunbird.org/api/course/v1/jobrequest/submit' \ --header 'x-authenticated-user-token: {{access_token}}' \ --header 'Authorization: Bearer {{jwt_token}}' \ --header 'Content-Type: application/json' \ --header 'x-channel-id: 01269878797503692810' \ --header 'x-authenticated-userid: 11f591e6-fea9-4ae1-a787-aea295c7ab8a' \ --data-raw '{ "request": { "tag": "program_6194f8d4db43ec4400a33b7c:01250894314817129555", "requestedBy": "fbe926ac-a395-40e4-a65b-9b4f711d7642", "dataset": "progress-exhaust", "datasetConfig": { "type": "program-user-exhaust", "params": { "filters": [ { "table_name": "program_enrollment", "table_filters": [ { "name": "program_id", "operator": "=", "value": "6194f8d4db43ec4400a33b7c" } ] }, { "table_name": "user_consent", "table_filters": [ { "name": "object_id", "operator": "=", "value": "6194f8d4db43ec4400a33b7c" }, { "name": "user_id", "operator": "=", "value": "13b24110-af97-430a-9d2c-0dd7ef6dccaa" } ] } ] }, "title": "User Detail Report" }, "output_format": "csv", "encryptionKey": "test1234" } }'

This curl worked by passing progress-exhaust as the dataset, But if we pass our program-user-exhaust as the dataset value it is throwing the above mentioned error.

Please help us out on this at the earliest... as this is blocking our dev testing as part of 5.3 Lern Release

Cc- @reshmi-nair @Hari-stackroute @AmiableAnil @aks30 @vijiurs

[Shakthieshwari]

@reshmi-nair Thanks for your help, As you told SubmitDataExhaustRequest API was not accepting our New ML PII Data Product Job_id = "program-exhaust" https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/kubernetes/opa/analytics/policies.rego#L60.

I have updated this line of code by adding our new job id and raised the PR project-sunbird/sunbird-devops#3833 to devops.

@anandp504 @manjudr Once devops merged I will let you know, Please deploy the changes or Let me know the steps to deploy the changes to be reflected in the SubmitDataExhaustRequest API in dev env

Cc- @Hari-stackroute @AmiableAnil @aks30 @vijiurs

[Shakthieshwari]

@anandp504 @manjudr Can you please approve this project-sunbird/sunbird-devops#3837 PR ?

Cc- @aks30 @vijiurs

[manjudr]

@Shakthieshwari I hope this issue has been resolved. If yes, can we close this?