How to implement Asynchronous Processing?

[jokiefer]

Recommendations from the JSON:API specs

JSON:API defines the following to handle async processing for long running tasks:

POST /photos HTTP/1.1

The request SHOULD return a status 202 Accepted with a link in the Content-Location header.

HTTP/1.1 202 Accepted
Content-Type: application/vnd.api+json
Content-Location: https://example.com/photos/queue-jobs/5234

{
 "data": {
   "type": "queue-jobs",
   "id": "5234",
   "attributes": {
     "status": "Pending request, waiting other process"
   },
   "links": {
     "self": "/photos/queue-jobs/5234"
   }
 }
}

-- json:api

My implementation to handle long running create endpoint

To follow that recommendation from the json:api specs if implemented it as following:

Long running create endpoint

from django_celery_results.models import TaskResult
from rest_framework_json_api.views import ModelViewSet

class OgcServiceViewSet(ModelViewSet):
    queryset = OgcService.objects.all()
    serializer_classes = {
        'default': OgcServiceSerializer,
        'create': OgcServiceCreateSerializer
    }

    filterset_fields = {
        'id': ('exact', 'lt', 'gt', 'gte', 'lte', 'in'),
        'title': ('icontains', 'iexact', 'contains'),
    }
    search_fields = ('id', 'title',)

    def get_serializer_class(self):
        return self.serializer_classes.get(self.action, self.serializer_classes['default'])

    def create(self, request, *args, **kwargs):
        serializer = self.get_serializer(data=request.data)
        serializer.is_valid(raise_exception=True)

        task = build_ogc_service.delay(data=serializer.data)
        task_result, created = TaskResult.objects.get_or_create(task_id=task.id)
        serialized_task_result = TaskResultSerializer(task_result)
        serialized_task_result_data = serialized_task_result.data
        # meta object is None... we need to set it to an empty dict to prevend uncaught runtime exceptions
        meta = serialized_task_result_data.get("meta", None)
        if not meta:
            serialized_task_result_data.update({"meta": {}})

        headers = self.get_success_headers(serialized_task_result_data)
        return Response(serialized_task_result_data, status=status.HTTP_202_ACCEPTED, headers=headers)

    def get_success_headers(self, data):
        try:
            return {'Content-Location': str(data[api_settings.URL_FIELD_NAME])}
        except (TypeError, KeyError):
            return {}

Create serializer with different field set

class OgcServiceCreateSerializer(ModelSerializer):

    # TODO: implement included serializer for ServiceAuthentication
    # included_serializers = {
    #     'auth': ServiceAuthentication,
    # }

    class Meta:
        model = OgcService
        fields = ("get_capabilities_url", )

TaskResult serializer

from django_celery_results.models import TaskResult
from registry.models.jobs import RegisterOgcServiceJob
from rest_framework_json_api.serializers import ModelSerializer

class TaskResultSerializer(ModelSerializer):

    class Meta:
        model = TaskResult
        fields = "__all__"

Response

The create enpoint is creating the celery job fine and response with the content like bellow:

Response Body

{
  "data": {
    "type": "OgcService",
    "id": "8",
    "attributes": {
      "task_id": "23e945e2-4c63-45bf-a76d-b64062db930e",
      "task_name": null,
      "task_args": null,
      "task_kwargs": null,
      "status": "PENDING",
      "worker": null,
      "content_type": "",
      "content_encoding": "",
      "result": null,
      "date_created": "2021-11-22T09:34:11.940341+01:00",
      "date_done": "2021-11-22T09:34:11.940404+01:00",
      "traceback": null,
      "meta": {}
    }
  }
}

Response headers

 allow: GET,POST,HEAD,OPTIONS  content-encoding: gzip  content-language: en  content-length: 238  content-type: application/vnd.api+json  referrer-policy: same-origin  server-timing: SQLPanel_sql_time;dur=3.191709518432617;desc="SQL 2 queries"  vary: Accept-Language,Cookie,Accept-Encoding,Origin  x-content-type-options: nosniff  x-frame-options: DENY 

Quesion

1. As you can see in the response body, the type differs to the resource type of the used serializer. How can dynamically set the type to the correct "type": "TaskResult" ?

2. How can i add url field to provide resource urls as meta information on the TaskResultSerializer without naming all fields explicit?

[sliverc]

I did not notice before that there is a recommendation from JSON:API spec on async calls... Great!

1. As far as I understand your example it would be enough if you set resource_name in the create serializer like the following.

class OgcServiceCreateSerializer:
  class Meta:
    resource_name = 'TaskResult'

2. I hope I understand you correctly here. DJA removes the URLs from fields as this is presented in the links (see here). You can write a custom metadata class which overwrites get_serializer_info though. To follow the JSON:API spec recommendation though you would need to overwrite get_links in your view to set the correct self link.

[jokiefer]

1. As far as I understand your example it would be enough if you set resource_name in the create serializer like the following.

That did not properly work. I defined the resource_name on OgcServiceCreateSerializer and/or on TaskResultSerializer. The endpoint will always response with:

{
  "errors": [
    {
      "detail": "The resource object's type (OgcService) is not the type that constitute the collection represented by the endpoint (TaskResult).",
      "status": "409",
      "source": {
        "pointer": "/data"
      },
      "code": "error"
    }
  ]
}

on given POST data:

{
	"data": {
		"attributes": {
			"get_capabilities_url": "http://geo5.service24.rlp.de/wms/karte_rp.fcgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities"
		},
		"relationships": {
			"owned_by_org": {
				"data": {
					"id": "b2fdc463-87cb-4deb-b39b-187f7616d1df",
					"type": "Organization"
				}
			}
		},
		"type": "OgcService"
	}
}

This is correct, cause JSON:API spec defines the following:

A server MUST return 409 Conflict when processing a POST request in which the resource object’s type is not among the type(s) that constitute the collection represented by the endpoint.

But i did not find a constraint in the JSON:API spec that does not allow on a endpoint to response with a different resource object type (TaskResult). Here is a discussion on the jsonapi.org forum, which handles about that.

So for my request the response looks like (with mismatching resource object type) :

{
  "data": {
    "type": "OgcService",
    "id": "9",
    "attributes": {
      "task_meta": null,
      "task_id": "d8146e14-d1ef-46d5-a6d2-be0de7017703",
      "task_name": null,
      "task_args": null,
      "task_kwargs": null,
      "status": "PENDING",
      "worker": null,
      "content_type": "",
      "content_encoding": "",
      "result": null,
      "date_created": "2021-11-24T09:12:35.056767+01:00",
      "date_done": "2021-11-24T09:12:35.056860+01:00",
      "traceback": null
    },
    "links": {
      "self": "http://testserver/api/v1/registry/task-results/9/"
    }
  }
}

Here is the link to the current implemented code: https://github.com/mrmap-community/mrmap/blob/a215b1c11bfff1f17d13c27d3e7b3ca88c3e3f87/backend/registry/api/views/service.py#L108

2. I hope I understand you correctly here. DJA removes the URLs from fields as this is presented in the links (see here). You can write a custom metadata class which overwrites get_serializer_info though. To follow the JSON:API spec recommendation though you would need to overwrite get_links in your view to set the correct self link.

OK - i understood.
My solution was it to add the field as HyperlinkedIdentityField:

class WebMapServiceSerializer(ModelSerializer):
    url = HyperlinkedIdentityField(
        view_name='registry:taskresult-detail',
    )
    class Meta:
        model = WebMapService
        fields = "__all__"

I struggled a bit with the view_name, cause i forgot to pass the app_name as well.

[sliverc]

True this error message makes sense. If we wanna support this async recommendation we properly would need a special case when response status code is 202 which than also handles Content-Location. I haven't investigated how this could be accomplished yet. Suggestions are welcome.

[jokiefer]

I think the handling of the resource_name and Content-Location could be done with a little switch in the renderer:

class JSONRenderer(renderers.JSONRenderer):
    ...
    def render(self, data, accepted_media_type=None, renderer_context=None):
        if serializer is not None:

            # Extract root meta for any type of serializer
            json_api_meta.update(self.extract_root_meta(serializer, serializer_data))

            if getattr(serializer, "many", False):
                ...
            else:
                ...
                if response.status_code == 202:
                    # handle async processing as recommended https://jsonapi.org/recommendations/#asynchronous-processing
                    resource_name = utils.get_resource_type_from_serializer(serializer)
                    response.headers.update({'Content-Location': serializer_data[api_settings.URL_FIELD_NAME]})

                resource_instance = serializer.instance
                json_api_data = self.build_json_resource_obj(
                    fields,
                    serializer_data,
                    resource_instance,
                    resource_name,
                    serializer,
                    force_type_resolution,
                )

[sliverc]

Hmm not so sure. During parsing wouldn't this still lead to the same error as when resource_name of the created serializer is set differently? Also in terms of features the renderer is already fairly complex and we need to work on that to make it easier and more maintainable. So I want to try to avoid making it more complex especially when adding optional features like async support. An idea (not thought through yet) but what about putting the support of async calls into a AsyncResponse class?

[sliverc]

Additionally as a workaround what you could do is set resource_name to False in your OgcServiceCreateSerializer. This means DJA rendering will be skipped and the data will be returned as you pass it on to Response. This means in your code you have to structure the data as it is required by the JSON:API async recommendation. Not ideal but should work as a workaround.