Merge pull request #320 from qld-gov-au/develop

CHDEXCHAE - De-identified enhancement, ckan 2.9.5-> 2.9.8

Linked Digital work on [CHDEXCHAE] plus much more

CKAN: [ckan-2.9.8-qgov.01](https://github.com/qld-gov-au/ckan/releases/tag/ckan-2.9.8-qgov.01)

Updates included but not limited to:
* ckanext-s3filestore to [0.7.7-qgov.6](https://github.com/qld-gov-au/ckanext-s3filestore/releases/tag/0.7.7-qgov.6)
* ckanext-harvest to [v1.4.2.a](https://github.com/qld-gov-au/ckanext-harvest/releases/tag/v1.4.2.a) (for including user agent in request to harvest external ckan instances)
* ckanext-odi-certificates [1.0.3](https://github.com/qld-gov-au/ckanext-odi-certificates/releases/tag/1.0.3)
* ckanext-qa [2.0.3-qgov.8](https://github.com/qld-gov-au/ckanext-qa/releases/tag/2.0.3-qgov.8)
* ckanext-datarequests [2.2.1-qgov.5 ](https://github.com/qld-gov-au/ckanext-datarequests/releases/tag/2.2.1-qgov.5)
* ckanext-resource-visibility [1.0.1](https://github.com/qld-gov-au/ckanext-resource-visibility/releases/tag/1.0.1) with Link Digital changes
* ckanext-validation [v0.0.8-qgov.7](https://github.com/qld-gov-au/ckanext-validation/releases/tag/v0.0.8-qgov.7) with Link Digital changes 
* ckanext-validation-schema-generator [1.0.2](https://github.com/qld-gov-au/ckanext-validation-schema-generator/releases/tag/1.0.2) with Link Digital changes
* ckanext-qgov  [5.0.7.a](https://github.com/qld-gov-au/ckanext-qgov/releases/tag/5.0.7.a)
* ckanext-data-qld  [7.0.0](https://github.com/qld-gov-au/ckanext-data-qld/releases/tag/7.0.0)
* ckanext-xloader [0.12.2](https://github.com/ckan/ckanext-xloader/releases/tag/0.12.2)

Infra:
* update cookbook [6.1.1](https://github.com/qld-gov-au/opswx-ckan-cookbook/releases/tag/6.1.1) to configure ckanext-resource-visibility cron jobs

.docker/test-OpenData.ini

@@ -99,7 +99,26 @@ ckan.redis.url = redis://redis:6379
 #       Add ``datapusher`` to enable DataPusher
 #       Add ``resource_proxy`` to enable resource proxying and get around the
 #       same origin policy
-ckan.plugins = stats text_view image_view recline_view datastore data_qld data_qld_google_analytics scheming_datasets validation dcat qa archiver report harvester_data_qld_geoscience harvest qgovext ytp_comments datarequests xloader csrf_filter resource_type_validation odi_certificates
+ckan.plugins =
+    stats text_view image_view recline_view
+    datastore validation
+    data_qld data_qld_google_analytics data_qld_integration
+    scheming_datasets
+    resource_type_validation
+    validation_schema_generator
+    dcat qa archiver report harvester_data_qld_geoscience harvest qgovext
+    ytp_comments datarequests xloader
+    csrf_filter
+    ssm_config odi_certificates
+    data_qld_test
+    resource_visibility
+
+## ckanext-data_qld
+ckanext.data_qld.resource_formats = CSV HTML JSON RDF TXT XLS
+ckanext.data_qld.reporting.de_identified_no_schema.count_from = 2022-01-01
+ckanext.data_qld.allow_bdd_test_plugin = True
+
+ckanext.ssm_config.region_name = ap-southeast-2
 
 # Define which views should be created by default
 # (plugins must be loaded in ckan.plugins)
@@ -114,7 +133,7 @@ ckan.views.default_views = image_view text_view recline_view
 #ckan.preview.image_formats = png jpeg jpg gif
 
 ## Front-End Settings
-ckan.favicon = https://www.qld.gov.au/favicon.ico
+ckan.favicon = /favicon.ico
 
 ## Internationalisation Settings
 ckan.locale_default = en_AU
@@ -188,15 +207,21 @@ ckanext.validation.run_on_create_async = True
 ckanext.validation.run_on_update_async = True
 ckanext.validation.run_on_create_sync = False
 ckanext.validation.run_on_update_sync = False
+
 ##Formats to validate
 ##By default validation will be run against the following formats: CSV, XLSX and XLS. You can modify these formats using the following option:
 ckanext.validation.formats = csv xlsx xls
 ##You can also provide validation options that will be used by default when running the validation:
 
 ## ckanext-scheming settings
 # see https://github.com/ckan/ckanext-scheming#configuration
-scheming.dataset_schemas = ckanext.data_qld:ckan_dataset.json ckanext.harvester_data_qld_geoscience:geoscience_dataset.json
-scheming.presets = ckanext.scheming:presets.json ckanext.data_qld:presets.json
+scheming.dataset_schemas =
+    ckanext.data_qld:ckan_dataset.json
+    ckanext.harvester_data_qld_geoscience:geoscience_dataset.json
+scheming.presets =
+    ckanext.scheming:presets.json
+    ckanext.data_qld:presets.json
+    ckanext.resource_visibility:schema/presets.json
 scheming.dataset_fallback = false
 
 ## ckanext-datarequests settings

.github/workflows/test-staging.yml

@@ -1,7 +1,8 @@
 ---
 #based on https://raw.githubusercontent.com/ckan/ckanext-scheming/master/.github/workflows/test.yml
 # alternative https://github.com/ckan/ckan/blob/master/contrib/cookiecutter/ckan_extension/%7B%7Bcookiecutter.project%7D%7D/.github/workflows/test.yml
-name: Tests
+name: Tests Staging
+run-name: "${{ github.actor }} PR ${{ github.event.pull_request.title }} on ${{ github.ref_name }} verifying Staging environment configuration"
 on:
   pull_request:
     branches:
@@ -13,7 +14,7 @@ jobs:
     strategy:
       matrix:
         application: [OpenData, Publications]
-        python-version: [py2, py3]
+        python-version: [py3]
         target_environment: [STAGING]
       fail-fast: true
 
@@ -33,12 +34,25 @@ jobs:
         run: bin/build.sh
         timeout-minutes: 20
 
+      - name: Test lint
+        run: bin/test-lint.sh
+        timeout-minutes: 30
+
       - name: Test
         run: bin/test.sh
         timeout-minutes: 40
 
+      - name: Test BDD
+        run: bin/test-bdd.sh
+        timeout-minutes: 45
+
+      - name: Retrieve logs
+        if: always()
+        run: bin/get-logs.sh
+        timeout-minutes: 5
+
       - name: Retrieve screenshots
-        if: failure()
+        if: always()
         run: bin/process-artifacts.sh
         timeout-minutes: 1
 

.github/workflows/test.yml

@@ -1,5 +1,6 @@
 ---
-name: Tests
+name: Tests Dev
+run-name: "${{ github.actor }} on ${{ github.ref_name }} verifying Dev environment configuration"
 on:
   push:
   schedule:
@@ -10,7 +11,7 @@ jobs:
     strategy:
       matrix:
         application: [OpenData, Publications]
-        python-version: [py2, py3]
+        python-version: [py3]
         target_environment: [DEV]
       fail-fast: false
 
@@ -30,12 +31,25 @@ jobs:
         run: bin/build.sh
         timeout-minutes: 20
 
+      - name: Test lint
+        run: bin/test-lint.sh
+        timeout-minutes: 30
+
       - name: Test
         run: bin/test.sh
         timeout-minutes: 40
 
+      - name: Test BDD
+        run: bin/test-bdd.sh
+        timeout-minutes: 60
+
+      - name: Retrieve logs
+        if: always()
+        run: bin/get-logs.sh
+        timeout-minutes: 5
+
       - name: Retrieve screenshots
-        if: failure()
+        if: always()
         run: bin/process-artifacts.sh
         timeout-minutes: 1
 

bin/get-logs.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+##
+# Run tests in CI.
+#
+set -e
+echo "output logs"
+ahoy logs

bin/serve.sh

@@ -15,7 +15,7 @@ done
 
 . ${APP_DIR}/bin/activate
 if (which ckan > /dev/null); then
-    ckan -c ${CKAN_INI} run -r
+    ckan -c ${CKAN_INI} run -r -t
 else
     paster serve ${CKAN_INI}
 fi

bin/test-all.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+##
+# Run tests in CI.
+#
+set -e
+
+./test-lint.sh
+
+./test.sh ||  (ahoy logs; exit 1)
+
+./test-bdd.sh ||  (ahoy logs; exit 1)
+
+

bin/test-bdd.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+##
+# Run tests in CI.
+#
+set -e
+
+
+echo "==> Run BDD tests"
+ahoy test-bdd
+#|| (ahoy logs; exit 1)

bin/test-lint.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+##
+# Run tests in CI.
+#
+set -e
+
+echo "==> Lint code"
+ahoy lint
+

bin/test.sh

@@ -4,11 +4,5 @@
 #
 set -e
 
-echo "==> Lint code"
-ahoy lint
-
 echo "==> Run Unit tests"
 ahoy test-unit
-
-echo "==> Run BDD tests"
-ahoy test-bdd || (ahoy logs; exit 1)

dev-requirements.txt

@@ -8,7 +8,7 @@ flake8==3.8.3
 mock
 pika>=1.1.0,<1.3.0
 # update this for ckanext-harvest
-pyOpenSSL==21.0.0
+pyOpenSSL>=23.0.0
 pytest-ckan
 pyyaml==5.4.1
 splinter>=0.13.0,<0.17

test/features/admin_reporting.feature

@@ -72,3 +72,43 @@ Feature: AdminReporting
         When I click the link with text that contains "Department of Health Spend Data"
         Then I should see "Department of Health Spend Data"
         And I should see "Data and Resources"
+
+    @fixture.dataset_with_schema::name=de-identified-package-without-schema::default_data_schema=::owner_org=department-of-health::title=de-identified-package-without-schema::de_identified_data=YES
+    @fixture.create_resource_for_dataset_with_params::package_id=de-identified-package-without-schema
+    Scenario: As an admin user of my organisation, when I view my admin report, I can verify de-identified datasets without default data schema
+        Given "Organisation Admin" as the persona
+        When I log in
+        And I go to my reports page
+        And I click the link with text that contains "Admin Report"
+        And I press the element with xpath "//button[contains(string(), 'Show')]"
+
+        Then I should see an element with xpath "//tr[@id='de_identified_datasets_no_schema']/td[contains(@class, 'metric-title') and position()=1]/a[contains(@href, 'de_identified_datasets_no_schema?report_type=admin') and contains(string(), 'De-identified datasets without default data schema (post-')]"
+        And I should see an element with xpath "//tr[@id='de_identified_datasets_no_schema']/td[contains(@class, 'metric-data') and position()=2]/a[contains(@href, 'de_identified_datasets_no_schema?report_type=admin')]"
+
+        When I click the link with text that contains "De-identified datasets without default data schema"
+        Then I should see "Admin Report: De-identified datasets without data schema validation (post-"
+        And I should see "Department of Health"
+        And I should see "de-identified-package-without-schema"
+        When I click the link with text that contains "de-identified-package-without-schema"
+        Then I should see "de-identified-package-without-schema"
+        And I should see "Data and Resources"
+
+    @fixture.dataset_with_schema::name=package-with-pending-assessment-resource::owner_org=department-of-health
+    @fixture.create_resource_for_dataset_with_params::package_id=package-with-pending-assessment-resource::name=pending-assessment-resource::request_privacy_assessment=YES
+    Scenario: Organisation Admin views 'Pending privacy assessment' count in the admin report
+        Given "Organisation Admin" as the persona
+        When I log in
+        And I visit "dashboard/reporting?report_type=admin&organisation=department-of-health"
+        And I click the link with text that contains "Admin Report"
+        And I press the element with xpath "//button[contains(string(), 'Show')]"
+
+        Then I should see an element with xpath "//tr[@id='pending_privacy_assessment']/td[contains(@class, 'metric-title') and position()=1]/a[contains(@href, 'pending_privacy_assessment?report_type=admin') and contains(string(), 'Pending privacy assessment')]"
+        And I should see an element with xpath "//tr[@id='pending_privacy_assessment']/td[contains(@class, 'metric-data') and position()=2]/a[contains(@href, 'pending_privacy_assessment?report_type=admin')]"
+
+        When I click the link with text that contains "Pending privacy assessment"
+        Then I should see "Admin Report: Pending privacy assessment"
+        And I should see "Department of Health"
+        And I should see "Total number of resources: 1"
+        And I should see "pending-assessment-resource"
+        Then I click the link with text that contains "pending-assessment-resource"
+        And I should see an element with xpath "//th[text()='Request privacy assessment']/following-sibling::td[text()='YES']"

test/features/data_qld_theme.feature

@@ -1,4 +1,4 @@
-Feature: Theme customisations
+Feature: Theme customisations (Publications and OpenData)
 
     @unauthenticated
     Scenario: Lato font is implemented on homepage
@@ -72,21 +72,23 @@ Feature: Theme customisations
         Then I should see "Password: Must contain at least one number, lowercase letter, capital letter, and symbol"
 
     @OpenData
+    @fixture.dataset_with_schema::name=package-with-csv-res::owner_org=test-organisation::[email protected]
+    @fixture.create_resource_for_dataset_with_params::package_id=package-with-csv-res::name=res-with-api-entry::xloader=True
     Scenario: As a publisher, when I create a resource with an API entry, I can download it in various formats
         Given "TestOrgEditor" as the persona
         When I log in
-        And I create a dataset with license "other-open" and "CSV" resource file "csv_resource.csv"
-        And I wait for 10 seconds
-        And I click the link with text that contains "Test Resource"
-        Then I should see an element with xpath "//a[contains(@class, 'resource-btn') and contains(@href, '/download/csv_resource.csv') and contains(string(), '(CSV)')]"
+        And I go to dataset "package-with-csv-res"
+        And I click the link with text that contains "res-with-api-entry"
+        And I reload page every 3 seconds until I see an element with xpath "//button[@data-toggle='dropdown']" but not more than 5 times
+        Then I should see an element with xpath "//a[contains(@class, 'resource-btn') and contains(@href, '/download/test.csv') and contains(string(), '(csv)')]"
         When I press the element with xpath "//button[@data-toggle='dropdown']"
         Then I should see an element with xpath "//a[contains(@href, '/datastore/dump/') and contains(string(), 'CSV')]"
         Then I should see an element with xpath "//a[contains(@href, '/datastore/dump/') and contains(@href, 'format=tsv') and contains(string(), 'TSV')]"
         Then I should see an element with xpath "//a[contains(@href, '/datastore/dump/') and contains(@href, 'format=json') and contains(string(), 'JSON')]"
         Then I should see an element with xpath "//a[contains(@href, '/datastore/dump/') and contains(@href, 'format=xml') and contains(string(), 'XML')]"
 
     @Publications
-    Scenario: Menu items are present and correct
+    Scenario: Publications - Menu items are present and correct
         When I go to "/dataset"
         Then I should see an element with xpath "//li[contains(@class, 'active')]/a[contains(string(), 'Publication') and (@href='/dataset' or @href='/dataset/')]"
         And I should see an element with xpath "//li[not(contains(@class, 'active'))]/a[contains(string(), 'Standards') and @href='/dataset/publishing-standards-publications-qld-gov-au']"

test/features/data_validation.feature

@@ -2,11 +2,11 @@
 @OpenData
 Feature: Data Validation
 
-    Scenario Outline: As a sysadmin, admin and editor user of the dataset organisation I cannot see the '</> JSON' button
+    Scenario Outline: As a sysadmin, admin and editor user of the dataset organisation I can see the '</> JSON' button
        Given "<User>" as the persona
         When I log in
         And I visit "dataset/new_resource/annakarenina"
-        And I should not see an element with xpath "//a[contains(string(), '</> JSON')]"
+        And I should see an element with xpath "//*[@id='resource-schema-buttons']//a[contains(string(), 'JSON')]"
 
         Examples: Users
         | User              |

test/features/datarequest.feature

@@ -58,16 +58,12 @@ Feature: Data Request
         And I should see an element with the css selector "span.error-block" within 1 seconds
         And I should see "Description cannot be empty" within 1 seconds
 
-    Scenario Outline: Data request creator and Sysadmin can see a 'Close' button on the data request detail page for opened data requests
-        Given "<User>" as the persona
+    Scenario: Data request creator and Sysadmin can see a 'Close' button on the data request detail page for opened data requests
+        Given "SysAdmin" as the persona
         When I log in and go to the data requests page
         And I press "Test Request"
         Then I should see an element with xpath "//a[contains(string(), 'Close')]"
 
-        Examples: Users
-        | User                  |
-        | SysAdmin              |
-
     Scenario Outline: Non admin users cannot see a 'Close' button on the data request detail page for opened data requests
         Given "<User>" as the persona
         When I log in and go to the data requests page

test/features/dataset_deletion.feature

@@ -14,26 +14,23 @@ Feature: Dataset deletion
         Then I fill in "author_email" with "[email protected]"
         Then I fill in "de_identified_data" with "NO" if present
         And I press the element with xpath "//form[contains(@class, 'dataset-form')]//button[contains(@class, 'btn-primary')]"
-        And I wait for 10 seconds
-        Then I execute the script "document.getElementById('field-image-url').value='https://example.com'"
+
+        Then I attach the file "csv_resource.csv" to "upload"
         Then I fill in "name" with "res1"
         Then I fill in "description" with "description"
         Then I fill in "size" with "1024" if present
         Then I fill in "resource_visibility" with "FALSE" if present
         Then I press the element with xpath "//button[@value='go-metadata']"
-        And I wait for 10 seconds
         Then I should see "Data and Resources"
 
         When I go to "/dataset/edit/dataset-deletion"
         Then I press the element with xpath "//a[@data-module='confirm-action']"
-        And I wait for 5 seconds
         Then I should see "Briefly describe the reason for deleting this dataset"
         And I should see an element with xpath "//div[@class='modal-footer']//button[@class='btn btn-primary' and @disabled='disabled']"
-        When I type "it should be longer than 10 characters" to "deletion_reason"
+        When I fill in "deletion_reason" with "it should be longer than 10 characters"
         Then I should not see an element with xpath "//div[@class='modal-footer']//button[@class='btn btn-primary' and @disabled='disabled']"
         Then I press the element with xpath "//div[@class='modal-footer']//button[@class='btn btn-primary']"
-        And I wait for 5 seconds
-        Then I should see "Dataset has been deleted"
+        And I reload page every 2 seconds until I see an element with xpath "//div[contains(@class, "alert") and contains(text(), "Dataset has been deleted")]" but not more than 5 times
         And I should not see "Dataset deletion"
         When I go to "/ckan-admin/trash"
         Then I should see "Dataset deletion"