From 69b612e4d81d3f9bf93a7bb9e9ffc0759c4d2019 Mon Sep 17 00:00:00 2001 From: Andreas Bisgaard Date: Fri, 22 Nov 2024 10:06:22 +0100 Subject: [PATCH] added extractor info --- Gemfile | 1 + Gemfile.lock | 3 +++ README.md | 42 ------------------------------ _config.yml | 3 +++ developing.md | 2 +- extractors.md | 7 ----- extractors/clevertrack.md | 30 ++++++++++++++++++++++ extractors/extractors.md | 32 +++++++++++++++++++++++ extractors/fleetcomplete.md | 29 +++++++++++++++++++++ extractors/gamfleet.md | 30 ++++++++++++++++++++++ extractors/mileagebook.md | 28 ++++++++++++++++++++ extractors/puma.md | 36 ++++++++++++++++++++++++++ extractors/skyhost.md | 51 +++++++++++++++++++++++++++++++++++++ index.md | 6 ++--- 14 files changed, 247 insertions(+), 53 deletions(-) delete mode 100644 README.md delete mode 100644 extractors.md create mode 100644 extractors/clevertrack.md create mode 100644 extractors/extractors.md create mode 100644 extractors/fleetcomplete.md create mode 100644 extractors/gamfleet.md create mode 100644 extractors/mileagebook.md create mode 100644 extractors/puma.md create mode 100644 extractors/skyhost.md diff --git a/Gemfile b/Gemfile index 157e918..fb3c645 100644 --- a/Gemfile +++ b/Gemfile @@ -5,3 +5,4 @@ gem "jekyll", "~> 4.3.4" # installed by `gem jekyll` gem "just-the-docs", "0.10.0" # pinned to the current release # gem "just-the-docs" # always download the latest release +gem "jekyll-relative-links", "0.7.0" diff --git a/Gemfile.lock b/Gemfile.lock index e521cf4..69773c7 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -39,6 +39,8 @@ GEM webrick (~> 1.7) jekyll-include-cache (0.2.1) jekyll (>= 3.7, < 5.0) + jekyll-relative-links (0.7.0) + jekyll (>= 3.3, < 5.0) jekyll-sass-converter (3.0.0) sass-embedded (~> 1.54) jekyll-seo-tag (2.8.0) @@ -83,6 +85,7 @@ PLATFORMS DEPENDENCIES jekyll (~> 4.3.4) + jekyll-relative-links (= 0.7.0) just-the-docs (= 0.10.0) BUNDLED WITH diff --git a/README.md b/README.md deleted file mode 100644 index fa2d741..0000000 --- a/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# FleetOptimiser OS2 Documentation - -## ️ Getting started - -This is a *bare-minimum* template to create a [Jekyll][Jekyll] site that: - -- uses the [Just the Docs][Just the Docs] theme; -- can be built and published on [GitHub Pages][GitHub Pages]; -- can be built and previewed locally, and published on other platforms. - -[Browse documentation][Just the Docs] to learn more about how to use this theme. - -Documentation in OS2 products must follow the documentation criteria from the [Standard for Public Code](https://standard.publiccode.net/) - -1. [Document your codebase objectives](https://standard.publiccode.net/criteria/document-codebase-objectives.html) -2. [Document the Code](https://standard.publiccode.net/criteria/document-the-code.html) - -Following these criteria not only opens up our codebase and fosters collaboration and transparency, but also encourages more organizations to use and adapt our open source products. - -## Contributing -Feel free to contribute, suggest an issue or submit a pull request with your improvements to this template. - -## Licensing and Attribution - -This repository is licensed under the [MIT License][MIT License]. You are generally free to reuse or extend upon this code as you see fit; just include the original copy of the license (which is preserved when you "make a template"). While it's not necessary, we'd love to hear from you if you do use this template, and how we can improve it for future use! - -The deployment GitHub Actions workflow is heavily based on GitHub's mixed-party [starter workflows][starter workflows]. A copy of their MIT License is available in [actions/starter-workflows][actions/starter-workflows]. - ---- - -[Jekyll]: https://jekyllrb.com -[Just the Docs]: https://just-the-docs.github.io/just-the-docs/ -[GitHub Pages]: https://docs.github.com/en/pages -[GitHub Pages / Actions workflow]: https://github.blog/changelog/2022-07-27-github-pages-custom-github-actions-workflows-beta/ -[Bundler]: https://bundler.io -[use this template]: https://github.com/just-the-docs/just-the-docs-template/generate -[`jekyll-default-layout`]: https://github.com/benbalter/jekyll-default-layout -[`jekyll-seo-tag`]: https://jekyll.github.io/jekyll-seo-tag -[MIT License]: https://en.wikipedia.org/wiki/MIT_License -[starter workflows]: https://github.com/actions/starter-workflows/blob/main/pages/jekyll.yml -[actions/starter-workflows]: https://github.com/actions/starter-workflows/blob/main/LICENSE -[^1]: [It can take up to 10 minutes for changes to your site to publish after you push the changes to GitHub](https://docs.github.com/en/pages/setting-up-a-github-pages-site-with-jekyll/creating-a-github-pages-site-with-jekyll#creating-your-site). diff --git a/_config.yml b/_config.yml index 7054112..2d6b253 100644 --- a/_config.yml +++ b/_config.yml @@ -8,3 +8,6 @@ logo: /assets/fologos.png aux_links: FleetOptimiser OS2: https://github.com/OS2fleetoptimiser/OS2fleetoptimiser + +plugins: + - jekyll-relative-links \ No newline at end of file diff --git a/developing.md b/developing.md index 85ce5b2..d5a67cb 100644 --- a/developing.md +++ b/developing.md @@ -128,5 +128,5 @@ docker run -p 6379:6379 redis -d npm run dev ``` -See [production](production.html) for environment variables for the backend. +See [production](production.md) for environment variables for the backend. diff --git a/extractors.md b/extractors.md deleted file mode 100644 index 877833d..0000000 --- a/extractors.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Extractors -layout: default -nav_order: 4 ---- - -# Extractors \ No newline at end of file diff --git a/extractors/clevertrack.md b/extractors/clevertrack.md new file mode 100644 index 0000000..9b024da --- /dev/null +++ b/extractors/clevertrack.md @@ -0,0 +1,30 @@ +--- +title: Clevertrack +layout: default +parent: Extractors +has_children: false +--- +# Clevertrack + +Clevertrack uses a method of patching already read trips. +If one does not patch you would continuously have to load already _known_ trips, because it's not possible to query +based off of vehicle id. Hence, you have to define a date from which you want to load trips, and manage patching. + +The connection requires a `TOKEN` environment which is a classic uid format. +In addition, you should provide a pickle path to; +- `CAR_TRIPS_IDS` : path to a pickle that stores mapping between vehicle ids and patched trips +- `PATCHED_IDS` : path to a pickle that stores record of patched trips + + +Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor: + +| function | description | +|------------------|--------------------------------------| +| set-vehicles | syncs the vehicles | +| set-roundtrips | loads, aggregate and save roundtrips | +| patch-roundtrips | patches read trips on the connection | + + +``` +["/bin/sh", "-c", "python -m fleetmanager.extractors.clevertrack set-vehicles && python -m fleetmanager.extractors.clevertrack set-roundtrips && python -m fleetmanager.extractors.clevertrack patch-roundtrips"] +``` diff --git a/extractors/extractors.md b/extractors/extractors.md new file mode 100644 index 0000000..3cb8451 --- /dev/null +++ b/extractors/extractors.md @@ -0,0 +1,32 @@ +--- +title: Extractors +layout: default +nav_order: 4 +has_children: true +--- + +# Extractors +The extractors are an important part of FleetOptimiser, which asserts that data is synced from the fleetmanagement systems. +The extractor will synchronise available metadata with vehicles, and will delta load the latest trips. +Trips will be aggregated from single GPS logs, to an aggregated format that will only contain coordinates of a saved start location, +such that the system does not save sensitive information. + + +The amount of exposed metadata differ from system to system, so with some integrations there are more sanitation of data. + + +All extractors expect the following env variables: + +| env | description | +|-----|-------------| +| DB_NAME | name of the database | +| DB_USER | user of the database | +| DB_PASSWORD | password of the user on the database | +| DB_URL | host and port to the database | +| DB_SERVER | dialect and driver | +| TRIP_DURATION | maximum duration of a roundtrip | +| MAX_DATE | first delta load date | +| DESCRIPTION_FIELDS | additional fields to save from APIs + + + diff --git a/extractors/fleetcomplete.md b/extractors/fleetcomplete.md new file mode 100644 index 0000000..84f673c --- /dev/null +++ b/extractors/fleetcomplete.md @@ -0,0 +1,29 @@ +--- +title: FleetComplete +layout: default +parent: Extractors +has_children: false +--- +# FleetComplete + +The FleetComplete extractor expects the connection key to be present in the env as a single string, like: + +``` +KEYS=keyToFleetComplete +``` + + +Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor: + +| function | description | +|------------------| ----------- | +| set-starts | loads and saves new locations | +| set-vehicles | syncs the vehicles | +| set-roundtrips | loads, aggregate and save roundtrips | +| clean-roundtrips | deletes X (keep_data in DB) month old roundtrips | + + + +``` +entrypoint: ["/bin/sh", "-c", "python -m fleetmanager.extractors.fleetcomplete set-starts && python -m fleetmanager.extractors.fleetcomplete set-vehicles && python -m fleetmanager.extractors.fleetcomplete set-roundtrips && python -m fleetmanager.extractors.fleetcomplete clean-roundtrips"] +``` \ No newline at end of file diff --git a/extractors/gamfleet.md b/extractors/gamfleet.md new file mode 100644 index 0000000..379b6e1 --- /dev/null +++ b/extractors/gamfleet.md @@ -0,0 +1,30 @@ +--- +title: Gamfleet +layout: default +parent: Extractors +has_children: false +--- +# Gamfleet + +You must modify the urls in the Gamfleet extractor code in order to point to the correct endpoints. + +To make the load more effective, we use a `LOAD_RECORD_PATH` to store load history of each vehicle. +This asserts that we don't load _old_ unusable trips. + +- `LOAD_RECORD_PATH` : path to json history (optional) +- `KEY` : key to the Gamfleet API + +Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor: + +| function | description | +|------------------| ----------- | +| set-starts | loads and saves new locations | +| set-vehicles | syncs the vehicles | +| set-roundtrips | loads, aggregate and save roundtrips | +| clean-roundtrips | deletes X (keep_data in DB) month old roundtrips | + + + +``` +entrypoint: ["/bin/sh", "-c", "python -m fleetmanager.extractors.gamfleet set-starts && python -m fleetmanager.extractors.gamfleet set-vehicles && python -m fleetmanager.extractors.gamfleet set-roundtrips && python -m fleetmanager.extractors.gamfleet clean-roundtrips"] +``` \ No newline at end of file diff --git a/extractors/mileagebook.md b/extractors/mileagebook.md new file mode 100644 index 0000000..f4bcdde --- /dev/null +++ b/extractors/mileagebook.md @@ -0,0 +1,28 @@ +--- +title: MileageBook +layout: default +parent: Extractors +has_children: false +--- +# MileageBook + +The MileageBook extractor expects the connection key as environment variable: + +``` +KEY=apiKey +``` + + +Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor: + +| function | description | +|------------------| ----------- | +| set-starts | loads and saves new locations | +| set-vehicles | syncs the vehicles | +| set-roundtrips | loads, aggregate and save roundtrips | +| clean-roundtrips | deletes X (keep_data in DB) month old roundtrips | + + +``` +entrypoint: ["/bin/sh", "-c", "python -m fleetmanager.extractors.fleetcomplete set-starts && python -m fleetmanager.extractors.fleetcomplete set-vehicles && python -m fleetmanager.extractors.fleetcomplete set-roundtrips && python -m fleetmanager.extractors.fleetcomplete clean-roundtrips"] +``` diff --git a/extractors/puma.md b/extractors/puma.md new file mode 100644 index 0000000..5e27739 --- /dev/null +++ b/extractors/puma.md @@ -0,0 +1,36 @@ +--- +title: Puma +layout: default +parent: Extractors +has_children: false +--- +# Puma + +PUMA connection requires VPN access, which details will not be explained here. +In addition to VPN access, it's also required to have read access to the PUMA database. + +We use a load record path to make loading of trips more efficient. In order not to continuously load unused _old_ trips. + +- `LOAD_RECORD_PATH` : path to a json mapping that stores load history +- `FORVALTNINGER` : list of forvaltninger to load trips from +- `PUMA_DRIVER` : the dialect of puma connection +- `PUMA_USER` : user on the puma connection +- `PUMA_URL` : url to the puma connection +- `PUMA_NAME` : name of the puma connection +- `PUMA_PASSWORD` : password to the puma connection + + +Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor: + +| function | description | +|------------------| ----------- | +| set-starts | loads and saves new locations | +| set-vehicles | syncs the vehicles | +| set-roundtrips | loads, aggregate and save roundtrips | +| clean-roundtrips | deletes X (keep_data in DB) month old roundtrips | + + +``` +entrypoint: ["/bin/sh", "-c", "python -m fleetmanager.extractors.puma set-starts && python -m fleetmanager.extractors.puma set-vehicles && python -m fleetmanager.extractors.puma set-roundtrips && python -m fleetmanager.extractors.puma clean-roundtrips"] +``` + diff --git a/extractors/skyhost.md b/extractors/skyhost.md new file mode 100644 index 0000000..3206c22 --- /dev/null +++ b/extractors/skyhost.md @@ -0,0 +1,51 @@ +--- +title: SkyHost +layout: default +parent: Extractors +has_children: false +--- +# SkyHost + +First version extracts data through a SOAP API. The environment must contain variable `KEYS` with value string that evaluates to list of strings e.g. + +``` +KEYS="['firstKey', 'secondKey']" +``` + +Use the `docker-compose-extractor.yaml` file and define the entrypoints function you want to use + +| function | description | +| -------- | ----------- | +| set-trackers | syncs the vehicles | +| set-roundtrips | loads, aggregate and save roundtrips | +| clean-roundtrips | deletes X (keep_data in DB) month old roundtrips | + +```` +entrypoint: ["/bin/sh", "-c", "python -m fleetmanager.extractors.skyhost set-trackers && python -m fleetmanager.extractors.skyhost set-roundtrips && python -m fleetmanager.extractors.skyhost clean-roundtrips"] +```` + + +# SkyHost V2 + +Second version uses the rest api, and uses env variables `ACCOUNT_IDS` and `API_KEYS` as a comma-separated list. +Each connection has exactly one account id and one api key. E.g. + +``` +ACCOUNT_IDS="accountId#1,accountId#2" +API_KEYS="apiKey#1,apiKey#2" +``` + +Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor: + +| function | description | +|--------------------| ----------- | +| set-allowed-starts | loads and saves new locations | +| set-trackers | syncs the vehicles | +| set-roundtrips | loads, aggregate and save roundtrips | +| clean-roundtrips | deletes X (keep_data in DB) month old roundtrips | + + +```` +entrypoint: ["/bin/sh", "-c", "python -m fleetmanager.extractors.skyhost set-allowed-starts && python -m fleetmanager.extractors.skyhost set-trackers && python -m fleetmanager.extractors.skyhost set-roundtrips && python -m fleetmanager.extractors.skyhost clean-roundtrips"] +```` + diff --git a/index.md b/index.md index b05b0b8..81140b5 100644 --- a/index.md +++ b/index.md @@ -19,6 +19,6 @@ Running FleetOptimiser in developer mode is easy and can be done in multiple way In addition to the description of how to run FleetOptimiser, we also offer extraction modules to fleet management systems that loads and aggregate driving-data. For more details go the [Extractors] page. -[Extractors]: extractors.html -[Production]: production.html -[Developing]: developing.html +[Extractors]: extractors.md +[Production]: production.md +[Developing]: developing.md