Clean repo

.github/workflows/test.yml

@@ -0,0 +1,45 @@
+name: Test
+
+on: pull_request
+
+jobs:
+  lint:
+    name: Linting and formatting
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v3
+        name: Install Python
+        with:
+          python-version: "3.x"
+
+      - name: Install tox
+        run: python -m pip install tox
+
+      - name: Run test suite
+        run: python -m tox -p -e flake8,black,yamllint
+
+  test_tox:
+    name: Run full tests
+    needs: lint
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v3
+        name: Install Python
+        with:
+          python-version: "3.x"
+      - name: Install Node
+        uses: actions/setup-node@v3
+        with:
+          node-version: "latest"
+
+      - name: Install node modules
+        run: npm ci
+
+      - name: Install tox
+        run: python -m pip install tox
+
+      - name: Run test suite
+        run: python -m tox -e py3

.gitignore

@@ -1,3 +1,6 @@
 fuseki-docker/databases/*
 __pycache__/
 node_modules/
+.tox
+*.egg-info/
+.pytest_cache

MANIFEST.in

@@ -7,7 +7,7 @@ recursive-include tdd/jsonld *.js
 include ./scripts/*.py
 
 include app.py
-include README.md
+include *.md
 include Dockerfile
 include docker-compose.yml
 include tox.ini

README.md

@@ -1,44 +1,55 @@
-# Python based Thing Description Directory API compliant to: https://www.w3.org/TR/wot-discovery/
+# TDD API
 
-## Deploy locally with docker-compose
+A Python and SPARQL based Thing Description Directory API compliant to:
+https://www.w3.org/TR/wot-discovery/
 
-For a quick launch of the SPARQL endpoint and TDD API with docker-compose:
+To learn more about the routes and function of this server, see
+the [API documentation](doc/api.md).
 
-```bash
-chmod a+rwx fuseki-docker/configuration
-chmod a+rwx fuseki-docker/databases
-docker-compose build # builds api and sparqlendpoint
-docker-compose up # runs api and sparqlendpoint
-```
+## Configuration
 
-If you want to deploy only the TDD API using docker-compose and use an
-existing SPARQL endpoint then you should edit the `config.toml` file with the
-appropriate `SPARQLENDPOINT_URL` value. Then run only the api image.
-If the api image is already built you do not have to rebuild, relaunching it
-will use the new config.
+The TDD API can be configured using two methods.
 
-```bash
-docker-compose build api # builds the api image
-docker-compose run api # runs the api
-```
+1. Using environment variables prefixing the variable names by `TDD__`
+   (to avoid conflicts)
 
-## Deploy production
+   ```
+     export TDD__SPARQLENDPOINT_URL="http://my-new-sparql.endpoint/address"
+     export TDD__CHECK_SCHEMA=True
+   ```
 
-If you want to deploy production without using docker or docker-compose you can use
-the following commands:
+2. Editing the `config.toml` file using the direct name of the variable
 
-```bash
-pip install .[prod]
-export TDD__SPARQLENDPOINT_URL=<sparql endpoint url>
-export TDD__TD_JSONSCHEMA=tdd/data/td-json-schema-validation.json
-gunicorn -b 0.0.0.0:5000 app:app
-```
+   ```
+     SPARQLENDPOINT_URL="http://my-new-sparql.endpoint/address"
+     CHECK_SCHEMA=True
+   ```
 
-You can change the `-b` parameter if you want to deploy only for localhost
-access, or public access, or change the port deployment.
+Those two configurations can be mixed with a priority as follows:
 
-In this example we use the configuration using the environment variables but you can edit
-ths `config.toml` file instead if you prefer.
+1. If the value is set in an environment variable, use the environment variable value
+2. If not, and it is set in the `config.toml`, use the `config.toml` value
+3. If not, use the default value
+
+The `config.toml` file can also be used to define FLask server configuration (c.f.
+[documentation](https://flask.palletsprojects.com/en/2.1.x/config/#builtin-configuration-values)).
+
+- The **SPARQLENDPOINT_URL** variable is mandatory
+
+### Configuration variables
+
+| Variable name             | default value                             | description                                                                                                                                                    |
+| ------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [TDD__]TD_REPO_URL        | http://localhost:5000                     | The URL to access the TDD API server                                                                                                                           |
+| [TDD__]SPARQLENDPOINT_URL | http://localhost:3030/things              | The SPARQL endpoint URL                                                                                                                                        |
+| [TDD__]CHECK_SCHEMA       | False                                     | Define if TDD API will check the TDs regarding to the JSON-Schema and SHACL shapes                                                                             |
+| [TDD__]MAX_TTL            | None                                      | Integer, maximum time-to-live (in seconds) that a TD will be kept on the server (unlimited if None)                                                            |
+| [TDD__]MANDATE_TTL        | False                                     | Boolean value, if set to True, it will only upload TDs having a time-to-live (ttl) value. The server will send a 400 HTTP code if the TD does not contain one. |
+| [TDD__]LIMIT_BATCH_TDS    | 25                                        | Default limit of returned TDs by batch (used for pagination)                                                                                                   |
+| [TDD__]VIRTUOSO_ENDPOINT  | False                                     | Set to True if you use a Virtuoso Endpoint (see [virtuoso errors](docs/sparql-endpoints/virtuoso.md))                                                          |
+| [TDD__]TD_JSONSCHEMA      | ./tdd/data/td-json-schema-validation.json | The path to the file containing JSON-Schema to validate the TDs                                                                                                |
+| [TDD__]TD_ONTOLOGY        | ./tdd/data/td.ttl                         | The path to the file containing the TD OWL Ontology (only used for SHACL validation)                                                                           |
+| [TDD__]TD_SHACL_VALIDATOR | ./tdd/data/td-validation.ttl              | The path to the file containing the SHACL shapes (only used for SHACL validation)                                                                              |
 
 ## Deploy to develop on the API
 
@@ -60,45 +71,34 @@ source .venv/bin/activate
 Install the project and it dependencies in the virtual environment by running:
 
 ```bash
-pip install -e .
+pip install -e .[dev]
 ```
 
-### Deploy a Fuseki server locally
-
-You can either use a distant SPARQL server or use a SPARQL server locally.
+Install the JavaScript dependencies (the project relies on jsonld.js for JSON-LD framing)
 
-We propose to use Apache Jena Fuseki, which has a nice administration interface.
-Download the Fuseki projet (apache-jena-fuseki-X.Y.Z.zip) from
-https://jena.apache.org/download/index.cgi
-
-Then unzip the downloaded archive.
-To launch the server, in the apache-jena-fuseki-X.Y.Z folder, run
-
-```
-./fuseki-server
+```bash
+npm ci
 ```
 
-The server will run on http://localhost:3030.
-If you want to create the dataset with the right configuration, you can copy-paste
-`fuseki-docker/configuration/things.ttl` into `apache-jena-fuseki-X.Y.Z/run/configuration`
+### Deploy a SPARQL endpoint
 
-```
-cp fuseki-docker/configuration/things.ttl path/to/apache-jena-fuseki-X.Y.Z/run/configuration
-```
+The TDD API relies on a SPARQL endpoint as database connection.
+You need to set up one before you run the project.
 
-More documentation on Fuseki in this project is available in `doc/fuseki.md`
+In the [SPARQL endpoint documentation](doc/sparql-endpoints/README.md) we provide
+you with guidelines on how to set-up your SPARQL endpoint.
 
 ### Run the flask server
 
+First, set up your configuration (the SPARQL endpoint URL) (see [configuration](#configuration))
+if your SPARQL endpoint URL is not the default http://localhost:3030/things.
+
+Then run the flask server at the root of this project in your python virtual environment.
+
 ```bash
-export TDD__SPARQLENDPOINT_URL=<sparql endpoint url>
-export TDD__TD_JSONSCHEMA=tdd/data/td-json-schema-validation.json
 flask run
 ```
 
-You can edit the `config.toml` file to change the configuration instead of using
-environment variables if you prefer.
-
 ## Import data using script
 
 To import the TDs from a directory to your SPARQL endpoint using the proxy api, run:
@@ -114,78 +114,63 @@ follows:
 python scripts/import_all_plugfest.py /path/to/TDs/directory <WOT API URL>/things?check-schema=false
 ```
 
-To import snapshots bundle (discovery data) use the proper script as following:
+## Deploy locally with docker-compose
+
+For a quick launch of the SPARQL endpoint and TDD API with docker-compose:
 
 ```bash
-python scripts/import_snapshot.py /path/to/snapshots.json <WOT API URL>/things
+chmod a+rwx fuseki-docker/configuration
+chmod a+rwx fuseki-docker/databases
+docker-compose build # builds api and sparqlendpoint
+docker-compose up # runs api and sparqlendpoint
 ```
 
-The `check-schema` param also works on this route.
+If you want to deploy only the TDD API using docker-compose and use an
+existing SPARQL endpoint then you should edit the `config.toml` file with the
+appropriate `SPARQLENDPOINT_URL` value (see [configuration](#configuration)).
+Then run only the api image.
+If the api image is already built you do not have to rebuild, relaunching it
+will use the new config.
 
-## Configuration
+```bash
+docker-compose build api # builds the api image
+docker-compose run api # runs the api
+```
 
-The TDD API can be configured using two methods. The first one is editing the
-`config.toml` file and the other one is using environment variables. Those two
-configuration can be mixed with a priority for the environment variables. It
-means that, for each variable, TDD API will search for the environment
-variables first, if they are not defined, then it will search for the
-`config.toml` values and if the variables are not defined in environment
-variable nor in `config.toml` the default value will be used.
+## Deploy production
 
-The configuration variables are the same on both methods, except that
-the environment variables must be prefixed with `TDD__` to avoid conflicts.
-The `config.toml` file can also be used to define FLask server configuration (c.f.
-[documentation](https://flask.palletsprojects.com/en/2.1.x/config/#builtin-configuration-values)).
+If you want to deploy production without using docker or docker-compose you can use
+the following commands:
 
-### Configuration variables
+```bash
+pip install .[prod]
+gunicorn -b 0.0.0.0:5000 app:app
+```
 
-| Variable name                 | default value                                 | description                                                                                                                                                    |
-| ----------------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [TDD__]TD_REPO_URL        | http://localhost:5000                         | The URL to access the TDD API server                                                                                                                           |
-| [TDD__]SPARQLENDPOINT_URL | http://localhost:3030/things                  | The SPARQL endpoint URL                                                                                                                                        |
-| [TDD__]TD_JSONSCHEMA      | ./tdd/data/td-json-schema-validation.json | The path to the file containing JSON-Schema to validate the TDs                                                                                                |
-| [TDD__]CHECK_JSON_SCHEMA  | True                                          | Define if TDD API will check the TDs regarding to the `TD_JSONSCHEMA` schema                                                                                   |
-| [TDD__]MAX_TTL            | None                                          | Integer, maximum time-to-live (in seconds) that a TD will be kept on the server (unlimited if None)                                                            |
-| [TDD__]MANDATE_TTL        | False                                         | Boolean value, if set to True, it will only upload TDs having a time-to-live (ttl) value. The server will send a 400 HTTP code if the TD does not contain one. |
+You can change the `-b` parameter if you want to deploy only for localhost
+access, or public access, or change the port deployment.
 
+In this example we use the configuration using the environment variables but you can edit
+the `config.toml` file instead if you prefer.
 
-## Notes on Virtuoso - TODO Change to a general section about tested Triplestores (Jena, GraphDB, Virtuoso and include this as a subsection)
-https://vos.openlinksw.com/owiki/wiki/VOS
+## Code quality
 
-`/Applications/Virtuoso Open Source Edition v7.2.app/Contents/virtuoso-opensource/bin` -> `./virtuoso-t +foreground +configfile ../database/virtuoso.ini`
+A few quality tests (formatting, linting, testing) have been developed to keep
+this code maintainable.
+Some tests have been developed to test the API's behaviour in python.
+We invite you to modify the tests if you change the behaviour and add
+new tests if you develop new features.
 
-`/Applications/Virtuoso\ Open\ Source\ Edition\ v7.2.app/Contents/virtuoso-opensource/bin/isql localhost:1111 -U dba -P dba`
-```
-GRANT EXECUTE ON DB.DBA.SPARQL_INSERT_DICT_CONTENT TO "SPARQL";
-GRANT EXECUTE ON DB.DBA.SPARQL_DELETE_DICT_CONTENT TO "SPARQL";
-DB.DBA.RDF_DEFAULT_USER_PERMS_SET ('nobody', 7);
-GRANT EXECUTE ON DB.DBA.SPARUL_RUN TO "SPARQL";
-GRANT EXECUTE ON DB.DBA.SPARQL_INSERT_QUAD_DICT_CONTENT TO "SPARQL";
-GRANT EXECUTE ON DB.DBA.L_O_LOOK TO "SPARQL";
-GRANT EXECUTE ON DB.DBA.SPARUL_CLEAR TO "SPARQL";
-GRANT EXECUTE ON DB.DBA.SPARUL_DROP TO "SPARQL";
-GRANT EXECUTE ON DB.DBA.SPARQL_UPDATE TO "SPARQL";
+You can do it with tox if you have it installed:
+
+```bash
+tox -p all
 ```
 
-http://127.0.0.1:8890/sparql   
-http://127.0.0.1:8890/conductor
-
-| User Name	| Default Password  | Usage |
-| :---  	| :---              | :--- |
-| dba	    | dba	            | Default Database Administrator account. |
-| dav	    | dav	            | WebDAV Administrator account. |
-| vad	    | vad	            | WebDAV account for internal usage in VAD (disabled by default). |
-| demo	    | demo	            | Default demo user for the demo database. This user is the owner of the Demo catalogue of the demo database. |
-| soap	    | soap	            | SQL User for demonstrating SOAP services. |
-| fori	    | fori	            | SQL user account for 'Forums' tutorial application demonstration in the Demo database. |
-
-Problem: Virtuoso 37000 Error SP031: SPARQL compiler: Blank node '_:b0' is not allowed in a constant clause   
-https://github.com/openlink/virtuoso-opensource/issues/126
-
-Go to the Virtuoso administration UI, i.e., http://host:port/conductor
-- Log in as user dba
-- Go to System Admin → User Accounts → Users
-- Click the Edit link
-- Set User type to SQL/ODBC Logins and WebDAV
-- From the list of available Account Roles, select SPARQL_UPDATE and click the >> button to add it to the right-hand list
-- Click the Save button
+Or manually if you do not have tox installed:
+
+```bash
+black .
+flake8
+pytests tests
+```