Merge pull request #21 from BritishGeologicalSurvey/fix-integration-t…

…ests

Fix integration tests

CONTRIBUTING.md

@@ -78,40 +78,43 @@ A Dockerised PostGIS version can be started with:
 
 ```bash
 export TEST_PG_PORT=5432
+export TEST_PG_PASSWORD=etlhelper_pw
 docker run -e POSTGRES_USER=etlhelper_user -e POSTGRES_DB=etlhelper \
-  -e POSTGRES_PASSWORD=etlhelper_pw --name etlhelper_postgis \
+  -e POSTGRES_PASSWORD=$TEST_PG_PASSWORD --name etlhelper_postgis \
   -d --rm -p $TEST_PG_PORT:5432 mdillon/postgis:11-alpine
 ```
 
 Tests are run with:
 
 ```bash
-export TEST_PG_PASSWORD=etlhelper_pw
 bash bin/run_tests_for_developer.sh
 ```
 
 The test-runner script will run tests within a dedicated container and provide
 HTML coverage output.  It can be viewed with `firefox htmlcov/index.html`.
 
 
-#### Additional integration tests
+#### Running additional BGS integration tests
 
 Additional integration tests can be run against internal BGS Oracle and SQL Server
 databases.
 The DbParams for these databases are defined by environment variables stored
 within the Continuous Integration system.
-The required environment variables to run the integration test suite can be
-seen in the `bin/run_tests_for_developers.sh` file.
-
-Internal BGS developers: `source` the `test_env_vars` Snippet from GitLab to
-configure local variables.
+To run these:
 
++ Go to Settings > CI / CD > Variables in GitLab
++ Use "Reveal values" and copy the contents of TEST_ENV_VARS
++ Paste into the terminal to set environment variables
++ Run tests as before
 
 ## Creating a new release
 
-Releases are created manually from the master branch with the following
-commands.
-The full integration test suite should be run before creating a release.
+Releases are created manually from the master branch via tags.
+This should be done via the GitHub web interface.
+The GitLab CI system will automatically perform the release following the next
+repository mirror synchronisation.
+
+The instructions below explain how to do a manual release.
 
 #### Tagging
 

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.6-slim
+FROM python:3.6.9-slim
 
 # Install package dependencies
 RUN apt-get update -y && \

etlhelper/db_params.py

@@ -15,14 +15,18 @@ class DbParams(dict):
     subclasses dict, to give dynamic attributes, following the pattern described
     here: https://amir.rachum.com/blog/2016/10/05/python-dynamic-attributes/
     """
-
     def __init__(self, dbtype='dbtype not set', **kwargs):
         kwargs.update(dbtype=dbtype.upper())
         super().__init__(kwargs)
         self.validate_params()
 
     def __getattr__(self, item):
-        return self[item]
+        try:
+            return self[item]
+        except KeyError:
+            # getattr should raise AttributeError, not KeyError
+            # https://docs.python.org/3/library/functions.html#getattr
+            raise AttributeError(f'No such attribute: {item}')
 
     def __dir__(self):
         return super().__dir__() + [str(k) for k in self.keys()]

test/conftest.py

@@ -21,7 +21,7 @@
     user='etlhelper_user')
 
 
-@pytest.fixture('module')
+@pytest.fixture(scope='module')
 def pgtestdb_insert_sql():
     """Return SQL command used to populate test database."""
     insert_sql = dedent("""
@@ -33,7 +33,7 @@ def pgtestdb_insert_sql():
     return insert_sql
 
 
-@pytest.fixture('function')
+@pytest.fixture(scope='function')
 def pgtestdb_test_tables(test_table_data, pgtestdb_conn, pgtestdb_insert_sql):
     """
     Create a table and fill with test data.  Teardown after the yield drops it

test/integration/db/test_oracle.py

@@ -69,7 +69,7 @@ def testdb_conn():
         yield conn
 
 
-@pytest.fixture('function')
+@pytest.fixture(scope='function')
 def test_tables(test_table_data, testdb_conn):
     """
     Create a table and fill with test data.  Teardown after the yield drops it