Task/1 Added functionality to process XML so we can extract data from En Tur

[tsunamiCam]

What was done

• Added functionality to process XML payloads
• Added configuration payload_type that can be json or xml. Defaults to json but we need to use xml for the En Tur API
• Added som details to the readme, introducing the so-called kantega variant.

Quick start - Extract En Tur data

By invoking the plugin directly

Clone the plugin to your local machine:

git clone https://github.com/tsunamiCam/tap-rest-api-msdk.git
cd tap-rest-api-msdk

Install and activate the Python envrionnment environment:

poetry install
poetry env

Run the tap using the sample config for the En Tur API:

tap-rest-api-msdk --config=config_sample_entur.json > data_out.json

An example config for En Tur:

{
    "api_url": "https://api.entur.io/realtime/v1/rest",
    "store_raw_json_message": false,
    "pagination_request_style": "header_link_paginator",
    "payload_type": "xml",
    "streams": [
        {
            "name": "vm",
            "path": "/vm",
            "records_path": "$.Siri.ServiceDelivery.VehicleMonitoringDelivery.VehicleActivity.[*]",
            "params": {
                "datasetId": "ATB",
                "requestorId": "f21a9317-9e15-4212-abe1-438d1be7a6d0"            
            }
        }
    ]
}

Run via Meltano

Add the following to your meltano.yml

version: 1
default_environment: dev
project_id: 31045931-05e8-4b07-96a2-6be2482a34be
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: git+https://github.com/tsunamiCam/tap-rest-api-msdk@kantega-variant
    config:
      api_url: https://api.entur.io/realtime/v1/rest
      store_raw_json_message: false
      pagination_request_style: header_link_paginator
      streams:
      - name: vm
        path: /vm
        records_path: $.Siri.ServiceDelivery.VehicleMonitoringDelivery.VehicleActivity.[*]
        params:
          datasetId: ATB

⚠️ Note that this is not an official variant. Instead we use the branch kantega-variant from from the fork at https://github.com/tsunamiCam/tap-rest-api-msdk.

Install the plugin:

meltano install extractor tap-rest-api-msdk

Invoke and save data to a file:

meltano invoke tap-rest-api-msdk > data_out.json