Merge pull request #42 from telefonicaid/feature/dynamic-external-url…

…-data-loading Possibility to compose external URLs dynamically using date templates
telefonicaid · Jun 21, 2017 · f3f2445 · f3f2445
2 parents f4505b5 + 7f8a1b2
commit f3f2445
Show file tree

Hide file tree

Showing 11 changed files with 311 additions and 40 deletions.
diff --git a/doc/manuals/simulation-configuration-file.md b/doc/manuals/simulation-configuration-file.md
@@ -306,7 +306,29 @@ The simulation configuration file accepts the following JSON properties or entri
         * **json**: Optional. A boolean with the same semantics as used in the `request` NPM package. This is: if `true`, 1) `body` must be a JSON-serializable object, 2) the `Content-type: application/json` header is automatically added to the requests and 3) the response body is parsed as JSON.
         * **method**: Mandatory. The HTTP method to be used in the request made to the external source endpoint. Accepted values: `GET` and `POST`.
         * **retry**:  Optional. Retry politics to be applied in case an error happens when requesting the data from the external source. It is an object including a `times` (with a default default value of 5) property to set the number of retries and an additional `interval` property (with a default default value of 0) to set the delay in milliseconds between the retries.
-        * **url**: Mandatory. The URL where the requests for the external data should be sent to. An accepted `url` property is the following: `"url": "http://www.aemet.es/es/eltiempo/observacion/ultimosdatos_6156X_datos-horarios.csv?k=and&l=6156X&datos=det&w=0&f=temperatura&x=h24"`.
+        * **url**: Mandatory. The URL where the requests for the external data should be sent to. The `url` entry accepts date templates so the final URLs can be composed dynamically. An accepted `url` property is the following: `"url": "http://www.aemet.es/es/eltiempo/observacion/ultimosdatos_6156X_datos-horarios.csv?k=and&l=6156X&datos=det&w=0&f=temperatura&x=h24"`. The supported date templates include (obviously these templates can be combined in one concrete `url`):
+            * `{YY}`: This template is substituted by the final 2 digits of the current year. For example, in 2017 the `url` with value `"http://www.example.com/{YY}"` will be finally generated as `"http://www.example.com/17"`.
+            * `{YYYY}`: This template is substituted by all the digits which corresponds to the current year. For example, in 2017 the `url` with value `"http://www.example.com/{YYYY}"` will be finally generated as `"http://www.example.com/2017"`.
+            * `{MM}`: This template is substituted by the 2 digits corresponding to the current month including a `0` at the beginning when appropriate. For example, in June the `url` with value `"http://www.example.com/{MM}"` will be finally generated as `"http://www.example.com/06"`.
+            * `{M*}@en`: This template is subsituted by the whole name of the current month in uppercase in English. For example, in June the `url` with value `"http://www.example.com/{M*}@en"` will be finally generated as `"http://www.example.com/JUNE"`.
+            * `{M*}@es`: This template is substituted by the whole name of the current month in uppercase in Spanish. For example, in June the `url` with value `"http://www.example.com/{M*}@es"` will be finally generated as `"http://www.example.com/JUNIO"`.
+            * `{m*}@en`: This template is substituted by the whole name of the current month in lowercase in English. For example, in June the `url` with value `"http://www.example.com/{m*}@en"` will be finally generated as `"http://www.example.com/june"`.
+            * `{m*}@es`: This template is substituted by the whole name of the current month in lowercase in Spanish. For example, in June the `url` with value `"http://www.example.com/{M*}@es"` will be finally generated as `"http://www.example.com/junio"`.
+            * `{MMM}@en`: This template is substituted by the first 3 characters of the current month in uppercase in English. For example, in June the `url` with value `"http://www.example.com/{MMM}@en"` will be finally generated as `"http://www.example.com/JUN"`.
+            * `{MMM}@es`: This template is substituted by the first 3 characters of the current month in uppercase in Spanish. For example, in June the `url` with value `"http://www.example.com/{MMM}@es"` will be finally generated as `"http://www.example.com/JUN"`.
+            * `{mmm}@en`: This template is substituted by the first 3 characters of the current month in lowercase in English. For example, in June the `url` with value `"http://www.example.com/{mmm}@en"` will be finally generated as `"http://www.example.com/jun"`.
+            * `{mmm}@es`: This template is substituted by the first 3 characters of the current month in lowercase in Spanish. For example, in June the `url` with value `"http://www.example.com/{mmm}@es"` will be finally generated as `"http://www.example.com/jun"`.
+            * `{DD}`: This template is substituted by the current day of the month including a `0` at the beginning when appropriate. For example, on the 5th, the `url` with value `"http://www.example.com/{DD}"` will be finally generated as `"http://www.example.com/05"`.
+            * `{DD-1}`: This template is substituted by the day before of the current day of the month, including a `0` at the beginning when appropriate. For example, on the 5th, the `url` with value `"http://www.example.com/{DD}"` will be finally generated as `"http://www.example.com/04"`.
+            * `{D*}@en`: This template is substituted by the whole name of the current day of the week in English. For example, on Wednesday, the `url` with value `"http://www.example.com/{D*}@en"` will be finally generated as `"http://www.example.com/WEDNESDAY"`.
+            * `{D*}@es`: This template is substituted by the whole name of the current day of the week in Spanish. For example, on Wednesday, the `url` with value `"http://www.example.com/{D*}@es"` will be finally generated as `"http://www.example.com/MIÉRCOLES"`.
+            * `{DDD}@en`: This template is substituted by the first letters of the name of the current day of the week in uppercase in English. For example, on Wednesday, the `url` with value `"http://www.example.com/{DDD}@en"` will be finally generated as `"http://www.example.com/WED"`.
+            * `{DDD}@es`: This template is substituted by the first letters of the name of the current day of the week in uppercase in Spanish. For example, on Wednesday, the `url` with value `"http://www.example.com/{DDD}@es"` will be finally generated as `"http://www.example.com/MIÉ"`.
+            * `{ddd}@en`: This template is substituted by the first letters of the name of the current day of the week in lowercase in English. For example, on Wednesday, the `url` with value `"http://www.example.com/{ddd}@en"` will be finally generated as `"http://www.example.com/wed"`.
+            * `{ddd}@es`: This template is substituted by the first letters of the name of the current day of the week in lowercase in Spanish. For example, on Wednesday, the `url` with value `"http://www.example.com/{ddd}@es"` will be finally generated as `"http://www.example.com/mié"`.
+            * `{hh}`: This template is substituted by the current hour in 24-hours format including a `0` at the beginning when appropriate. For example, at 10 PM, the `url` with value `"http://www.example.com/{hh}"` will be finally generated as `"http://www.example.com/22"`.
+            * `{mm}`: This template is substituted by the current minutes including a `0` at the beginning when appropriate. For example, at 10:05 PM, the `url` with value `"http://www.example.com/{mm}"` will be finally generated as `"http://www.example.com/05"`.
+            * `{hh}`: This template is substituted by the current seconds including a `0` at the beginning when appropriate. For example, at 10:05:09 PM, the `url` with value `"http://www.example.com/{ss}"` will be finally generated as `"http://www.example.com/09"`.
 * **devices**: Information about the devices to be updated during this concrete simulation. The `devices` entries are just like the previous `entities` entries described above with the following modifications:
     1. Instead of the `entity_name`, a `device_id` has to be specified (in case the `count` property is used, the `device_id` property is set just like the `entity_name` as describe above in the `count` property description).
     2. A `protocol` property has to be set specifying the device protocol. Accepted values are: `UltraLight::HTTP`, `UltraLight::MQTT`, `JSON::HTTP` and `JSON::MQTT`.

diff --git a/...ples/urbo-simulation-external-source.json → ...rbo-simulation-external-source-aemet.json b/...ples/urbo-simulation-external-source.json → ...rbo-simulation-external-source-aemet.json
diff --git a/examples/urbo-simulation-external-source-air-quality.json b/examples/urbo-simulation-external-source-air-quality.json
@@ -0,0 +1,167 @@
+{
+  "require": ["tabletojson"],
+  "domain": {
+    "service": "service",
+    "subservice": "/subservice"
+  },
+  "contextBroker": {
+    "protocol": "http",
+    "host": "1.2.3.4",
+    "port": 1026,
+    "ngsiVersion": "1.0"
+  },
+  "authentication": {
+    "provider": "keystone",
+    "protocol": "http",
+    "host": "1.2.3.4",
+    "port": 5001,
+    "user": "user",
+    "password": "password",
+    "retry": {
+      "times": 10,
+      "interval": 5000
+    }
+  },
+  "entities": [
+    {
+      "external": {
+        "retry": {
+          "times": 10,
+          "interval": 1000
+        },
+        "method": "GET",
+        "url": "http://www.juntadeandalucia.es/medioambiente/atmosfera/informes_siva/{mmm}@es{YY}/nma{YY}{MM}{DD-1}.htm",
+        "headers": {
+          "Cache-Control": "no-cache"
+        },
+        "collector": "var tabletojson = require('tabletojson'); function getDate(dateString) { return new Date('20' + dateString.substring(6,8), dateString.substring(3,5) - 1, dateString.substring(0,2), dateString.substring(9,11), dateString.substring(12));} if (external.response.statusCode === 200) { var tables = tabletojson.convert(external.response.body); var result = []; for (var ii = 1; ii < tables[2].length - 1; ii++) { var entry = [{'name': 'address', 'type': 'Address', 'value': {'addressCountry': 'ES', 'addressLocality': tables[1][1][1], 'streetAddress': tables[1][3][1]}}, {'name': 'dateObserved', 'type': 'DateTime', 'value': getDate(tables[2][ii][0]).toISOString()}, {'name': 'S02', 'type': 'Number', 'value': tables[2][ii][1]}, {'name': 'PM10', 'type': 'Number', 'value': tables[2][ii][2]}, {'name': 'NO2', 'type': 'Number', 'value': tables[2][ii][3]}, {'name': 'CO2', 'type': 'Number', 'value': tables[2][ii][4]}, {'name': 'O3', 'type': 'Number', 'value': tables[2][ii][5]}, {'name': 'measurand', 'type': 'List', 'value': ['SO2,' + tables[2][ii][1] + ',GQ,Sulfur Dioxide', 'PM10,' + tables[2][ii][2] + ',GQ,Suspended Particles', 'NO2,' + tables[2][ii][3] + ',GQ,Nitrogen Dioxide', 'CO2,' + tables[2][ii][4] + ',GQ,Carbon Monoxide', 'O3,' + tables[2][ii][5] + ',GQ,Ozone']}]; result.push(entry); } module.exports = result; }"
+      },
+      "entity_type": "AirQualityObserved",
+      "entity_name": "malaga_airq_elatabal",
+      "schedule": "0 */1 * * * *",
+      "staticAttributes": [
+        {
+          "name": "location",
+          "type": "geo:json",
+          "value": {
+            "type": "Point",
+            "coordinates": [
+              -4.46553495908059,
+              36.7295574570493
+            ]
+          }
+        },
+        {
+          "name": "source",
+          "type": "URL",
+          "value": "http://www.juntadeandalucia.es"
+        }
+      ]
+    },
+    {
+      "external": {
+        "retry": {
+          "times": 10,
+          "interval": 1000
+        },
+        "method": "GET",
+        "url": "http://www.juntadeandalucia.es/medioambiente/atmosfera/informes_siva/{mmm}@es{YY}/nma{YY}{MM}{DD-1}.htm",
+        "headers": {
+          "Cache-Control": "no-cache"
+        },
+        "collector": "var tabletojson = require('tabletojson'); function getDate(dateString) { return new Date('20' + dateString.substring(6,8), dateString.substring(3,5) - 1, dateString.substring(0,2), dateString.substring(9,11), dateString.substring(12));} if (external.response.statusCode === 200) { var tables = tabletojson.convert(external.response.body); var result = []; for (var ii = 1; ii < tables[4].length - 1; ii++) { var entry = [{'name': 'address', 'type': 'Address', 'value': {'addressCountry': 'ES', 'addressLocality': tables[3][1][1], 'streetAddress': tables[3][3][1]}}, {'name': 'dateObserved', 'type': 'DateTime', 'value': getDate(tables[4][ii][0]).toISOString()}, {'name': 'S02', 'type': 'Number', 'value': tables[4][ii][1]}, {'name': 'PM10', 'type': 'Number', 'value': tables[4][ii][2]}, {'name': 'NO2', 'type': 'Number', 'value': tables[4][ii][3]}, {'name': 'CO2', 'type': 'Number', 'value': tables[4][ii][4]}, {'name': 'O3', 'type': 'Number', 'value': tables[4][ii][5]}, {'name': 'measurand', 'type': 'List', 'value': ['SO2,' + tables[4][ii][1] + ',GQ,Sulfur Dioxide', 'PM10,' + tables[4][ii][2] + ',GQ,Suspended Particles', 'NO2,' + tables[4][ii][3] + ',GQ,Nitrogen Dioxide', 'CO2,' + tables[4][ii][4] + ',GQ,Carbon Monoxide', 'O3,' + tables[4][ii][5] + ',GQ,Ozone']}]; result.push(entry); } module.exports = result; }"
+      },
+      "entity_type": "AirQualityObserved",
+      "entity_name": "malaga_airq_carranque",
+      "schedule": "0 */1 * * * *",
+      "staticAttributes": [
+        {
+          "name": "location",
+          "type": "geo:json",
+          "value": {
+            "type": "Point",
+            "coordinates": [
+              -4.44749849600678,
+              36.7196374482505
+            ]
+          }
+        },
+        {
+          "name": "source",
+          "type": "URL",
+          "value": "http://www.juntadeandalucia.es"
+        }
+      ]
+    },
+    {
+      "external": {
+        "retry": {
+          "times": 10,
+          "interval": 1000
+        },
+        "method": "GET",
+        "url": "http://www.juntadeandalucia.es/medioambiente/atmosfera/informes_siva/{mmm}@es{YY}/nma{YY}{MM}{DD-1}.htm",
+        "headers": {
+          "Cache-Control": "no-cache"
+        },
+        "collector": "var tabletojson = require('tabletojson'); function getDate(dateString) { return new Date('20' + dateString.substring(6,8), dateString.substring(3,5) - 1, dateString.substring(0,2), dateString.substring(9,11), dateString.substring(12));} if (external.response.statusCode === 200) { var tables = tabletojson.convert(external.response.body); var result = []; for (var ii = 1; ii < tables[6].length - 1; ii++) { var entry = [{'name': 'address', 'type': 'Address', 'value': {'addressCountry': 'ES', 'addressLocality': tables[5][1][1], 'streetAddress': tables[5][3][1]}}, {'name': 'dateObserved', 'type': 'DateTime', 'value': getDate(tables[6][ii][0]).toISOString()}, {'name': 'S02', 'type': 'Number', 'value': tables[6][ii][1]}, {'name': 'PM10', 'type': 'Number', 'value': tables[6][ii][2]}, {'name': 'NO2', 'type': 'Number', 'value': tables[6][ii][3]}, {'name': 'CO2', 'type': 'Number', 'value': tables[6][ii][4]}, {'name': 'O3', 'type': 'Number', 'value': tables[6][ii][5]}, {'name': 'measurand', 'type': 'List', 'value': ['SO2,' + tables[6][ii][1] + ',GQ,Sulfur Dioxide', 'PM10,' + tables[6][ii][2] + ',GQ,Suspended Particles', 'NO2,' + tables[6][ii][3] + ',GQ,Nitrogen Dioxide', 'CO2,' + tables[6][ii][4] + ',GQ,Carbon Monoxide', 'O3,' + tables[6][ii][5] + ',GQ,Ozone']}]; result.push(entry); } module.exports = result; }"
+      },
+      "entity_type": "AirQualityObserved",
+      "entity_name": "malaga_airq_campanillas",
+      "schedule": "0 */1 * * * *",
+      "staticAttributes": [
+        {
+          "name": "location",
+          "type": "geo:json",
+          "value": {
+            "type": "Point",
+            "coordinates": [
+              -4.56095497889189,
+              36.7278572726192
+            ]
+          }
+        },
+        {
+          "name": "source",
+          "type": "URL",
+          "value": "http://www.juntadeandalucia.es"
+        }
+      ]
+    },
+    {
+      "external": {
+        "retry": {
+          "times": 10,
+          "interval": 1000
+        },
+        "method": "GET",
+        "url": "http://www.juntadeandalucia.es/medioambiente/atmosfera/informes_siva/{mmm}@es{YY}/nma{YY}{MM}{DD-1}.htm",
+        "headers": {
+          "Cache-Control": "no-cache"
+        },
+        "collector": "var tabletojson = require('tabletojson'); function getDate(dateString) { return new Date('20' + dateString.substring(6,8), dateString.substring(3,5) - 1, dateString.substring(0,2), dateString.substring(9,11), dateString.substring(12));} if (external.response.statusCode === 200) { var tables = tabletojson.convert(external.response.body); var result = []; for (var ii = 1; ii < tables[10].length - 1; ii++) { var entry = [{'name': 'address', 'type': 'Address', 'value': {'addressCountry': 'ES', 'addressLocality': tables[9][1][1], 'streetAddress': tables[9][3][1]}}, {'name': 'dateObserved', 'type': 'DateTime', 'value': getDate(tables[10][ii][0]).toISOString()}, {'name': 'S02', 'type': 'Number', 'value': tables[10][ii][1]}, {'name': 'PM10', 'type': 'Number', 'value': tables[10][ii][2]}, {'name': 'NO2', 'type': 'Number', 'value': tables[10][ii][3]}, {'name': 'CO2', 'type': 'Number', 'value': tables[10][ii][4]}, {'name': 'O3', 'type': 'Number', 'value': tables[10][ii][5]}, {'name': 'measurand', 'type': 'List', 'value': ['SO2,' + tables[10][ii][1] + ',GQ,Sulfur Dioxide', 'PM10,' + tables[10][ii][2] + ',GQ,Suspended Particles', 'NO2,' + tables[10][ii][3] + ',GQ,Nitrogen Dioxide', 'CO2,' + tables[10][ii][4] + ',GQ,Carbon Monoxide', 'O3,' + tables[10][ii][5] + ',GQ,Ozone']}]; result.push(entry); } module.exports = result; }"
+      },
+      "entity_type": "AirQualityObserved",
+      "entity_name": "malaga_airq_campanillas",
+      "schedule": "0 */1 * * * *",
+      "staticAttributes": [
+        {
+          "name": "location",
+          "type": "geo:json",
+          "value": {
+            "type": "Point",
+            "coordinates": [
+              -4.56095497889189,
+              36.7278572726192
+            ]
+          }
+        },
+        {
+          "name": "source",
+          "type": "URL",
+          "value": "http://www.juntadeandalucia.es"
+        }
+      ]
+    }
+  ]
+}