Sharing bulk update api

[vietnguyen]

Requirement

• This issue is about supporting "bulk sharing" of metadata objects. This means the ability to add or remove many users and user groups to many objects in one API operation.
• This API should not support keeping metadata objects in sync over time, and instead treat it as a one-time operation.
• The API needs to respect the sharing access control, in that the current user must have access to edit the sharing of the objects being updated.
• The main use-case for this API is a web client (maintenance app) which will add or remove user/user groups to/from many metadata objects in bulk.

Endpoints

Using /api/{object-type}/sharing with PATCH request

• This endpoint allow user to send a Json Patch request to a single object-type which should be used by the maintenance app.
• Note that we still support JSON PATCH request for one object with endpoint api/{object-type}/{uid}. Without path variable uid the request will be treated as a bulk patch request.
  Example: api/dataElements/sharing

Using /api/metadata/sharing with PATCH request

• We will also support bulk PATCH request for multiple object-types in one payload. This could be a new feature in Metadata Import/Export app.
• This endpoint should be used by the import/export app.

Parameters

Name	Default	Description
atomic	false	If this is set to true, then the batch function will stop and not updating any objects if there is an error Otherwise, if this is false then the function will try to proceed with best effort mode.

Backend function

• Add new batch method to AbstractCrudController for serving PATCH request to api/{object-type}/
• Create a new class

class BulkJsonPatch {
  Map<Class<?>,List<String>> targetUids;
  JsonPatch patch;
}

• Get persisted objects and validate for existence.
• Apply patchOperations to persisted objects .
• Import patched objects using metadata import service.

Validation

• All object UID will be validated for existence.
• Current User need to have METADATA_WRITE permission on updating objects.
• All existing validations from metadata import service.

Response

• Response format should be same as from /api/metadata api.

Payload

• Payload for single object type using /api/{object-type}/sharing looks like this

{
  "dataSets":[
    "cYeuwXTCPkU",
    "aYeuwXTCPkU"
  ],
  "patch":[
    {
      "op":"add",
      "path":"/sharing/users/DXyJmlo9rge",
      "value":{
        "access":"rw------",
        "id":"DXyJmlo9rge"
      }
    },
    {
      "op":"remove",
      "path":"/sharing/users/N3PZBUlN8vq"
    }
  ]
}

• Payload for multiple object types in one payload using api/metadata/sharing

{
    "dataElements": [
        {
            "fbfJHSPpUQD": [
                {
                    "op": "replace",
                    "path": "/sharing/users",
                    "value": {
                        "NOOF56dveaZ": {
                            "access": "rw------",
                            "id": "CotVI2NX0rI"
                        },
                        "Kh68cDMwZsg": {
                            "access": "rw------",
                            "id": "DLjZWMsVsq2"
                        }
                    }
                }
            ]
        }
    ],
    "dataSets": [
        {
            "cYeuwXTCPkU": [
                {
                    "op": "add",
                    "path": "/sharing/users/DXyJmlo9rge",
                    "value": {
                        "access": "rw------",
                        "id": "DXyJmlo9rge"
                    }
                }
            ]
        },
        {
            "cYeuwXTCPkU": [
                {
                    "op": "remove",
                    "path": "/sharing/users/N3PZBUlN8vq"
                }
            ]
        }
    ],
    "programs": [
        {
            "GOLswS44mh8": [
                {
                    "op": "add",
                    "path": "/sharing/userGroups",
                    "value": {
                        "NOOF56dveaZ": {
                            "access": "rw------",
                            "id": "NOOF56dveaZ"
                        },
                        "Kh68cDMwZsg": {
                            "access": "rw------",
                            "id": "Kh68cDMwZsg"
                        }
                    }
                }
            ]
        }
    ]
}

Bonus

• Can be applied to other properties as well, such as attributeValues.

[jbee]

I think the other possible format we talked about would look something like this:

{
  "dataSets":[
    "cYeuwXTCPkU",
    "aYeuwXTCPkU"
  ],
  "patch":[
    {
      "op":"add",
      "path":"/sharing/users/DXyJmlo9rge",
      "value":{
        "access":"rw------",
        "id":"DXyJmlo9rge"
      }
    },
    {
      "op":"remove",
      "path":"/sharing/users/N3PZBUlN8vq"
    }
  ]
}

In java this could be modelled as BulkJsonPatch looking like this:

class BulkJsonPatch {
  Map<Class<?>,List<String>> targetUids;
  JsonPatch patch;
}

JSON to Java mapping could be done with help of @JsonAnySetter.

Plus:

• works for any/all bulk operations; single type of objects as well as mixed or multiple types of objects
• all API cases can be processed with a single bulk patch processing
• does not require extending the JsonPatchOperation hierarchy

Minus:

• same operation is applied to all target - does not allow for different operations on different targets

I think @mortenoh suggested something like the following to also allow different operations:

{
  "dataSets":[
    {
      "cYeuwXTCPkU":[
        {
          "op":"add",
          "path":"/sharing/users/DXyJmlo9rge",
          "value":{
            "access":"rw------",
            "id":"DXyJmlo9rge"
          }
        },
        {
          "op":"remove",
          "path":"/sharing/users/N3PZBUlN8vq"
        }
      ]
    },
    {
      "aYeuwXTCPkU":[...]
    }
  ],
  "dataElements":[...]
}

The outer wrapper then could be:

class BulkJsonPatch {
   Map<Class<?>, Map<String, JsonPatch>> patches;
}

I think this has all the benefits of the solution for same operation on all objects and does not have the drawback that it cannot model different changes on different objects. However this also means if what you really wanted is the same operation on all objects and those are lots of objects this payload gets very big compared to the same operation version. This could be an argument to allow even both variants.

[vietnguyen]

@jbee

• I think the "same operations for all objects" make more sense for the UI in Maintenance app, and the other format might be useful in Import app. Supporting both formats might be a good solution, doesn't require too much codes anyway.

• Just not sure about the name of the endpoint. Maybe api/bulkSharing ( same operations for all objects ) and api/sharing ?

• One question : should we validate the path in each of the JsonPatchOperation to make sure that all the patch operations only apply to sharing property ? Otherwise this endpoint could be misused to update other properties. But at the same time I think we might also want this function to work with other properties as well.

[jbee]

should we validate the path...

Sounds like a good idea.

On the path the nicest solution for the user would be to just have /sharing and to map based on the JSON structure given. But I understand this could be cumbersome and require fair deal of code. @mortenoh , @larshelge WDYT about the resource path?

[vietnguyen]

Maybe in 2.38 we can do this:

• api/{object-type}/sharing will support "same operations for all objects".
• api/metadata/sharing support the other format.

[jbee]

Yeah, almost sounds obvious to me now that you said it :D

[vietnguyen]

This has been implemented. Check this api document for more details.