ckanext-dgu - data.gov.uk extension

This is an extension to CKAN that provides customisations specifically for the data.gov.uk project.

The official version is available at: https://github.com/datagovuk/ckanext-dgu

Contributions from 1st March 2012 are by the Cabinet Office. It is Crown Copyright and opened up under both the Open Government Licence (OGL) (which is compatible with Creative Commons Attibution License) and the GNU Affero General Public License (AGPL) v3.0. Before 1st March 2012, contributions are Copyright (c) 2010-2012 Open Knowledge Foundation. This material is open and licensed under the GNU Affero General Public License (AGPL) v3.0.

OGL terms: http://www.nationalarchives.gov.uk/doc/open-government-licence/ AGPL terms: http://www.fsf.org/licensing/licenses/agpl-3.0.html

Related extensions that data.gov.uk use in addition are: ckanext-archiver, ckanext-harvest, ckanext-spatial, ckanext-os, ckanext-qa, ckanext-social

Plugins

This extension contains a number of elements, principally:

• DGU-specific templates
  • Custom dataset edit form - includes a number of custom fields such as temporal_coverage and geographic_coverage.
  • Harvest Object inserted into the CKAN package view page.
• DGU-customized search - different facets etc
• Reports (integrated from ckanext-report)
• Drupal integration - Drupal login, user details copied into CKAN, comments below datasets, provide APIs to Drupal for displaying recent datasets
• Authorization customizations - display of users is tightened up
• Feedback form (below datasets)
• gov_daily - a script (for running daily) that save the database dumps for end-users (JSON/CSV) and backups (SQL).
• various other command-line utilities

dgu_form

ckanext.dgu.plugin:DguForm
ckanext.dgu.controllers.package:PackageController

Based on PackageController but at /dataset/* rather than /package/* and adds the delete function. Proxy for getting Drupal comments (only for running in paster). NO FORM HERE

dgu_drupal_auth

ckanext.dgu.plugin:DrupalAuthPlugin
ckanext.dgu.authentication.drupal_auth:DrupalAuthMiddleware

Middleware to log-in the user based on Drupal cookies and requests to Drupal.

dgu_auth_api

ckanext.dgu.plugin:AuthApiPlugin
ckanext.dgu.authorize

Changes permissions:

• hierarchy structure - edit package/group if is an editor for the group or an admin for the group or its parents.
• All package creations/edits need an API key - no anonymous ones
• UKLP packages can't be edited through the form or API - only by harvesting (unless sysadmin)
• ONS packages can't be edited through the form or API - only by ONS loader (unless sysadmin)
• Packages can only be deleted by sysadmin or UKLP packages by their editor/admin.
• Users can only be viewed by the user and sysadmin
• User list can only be viewed by sysadmin, editors, admins.

dgu_publishers

ckanext.dgu.plugin:PublisherPlugin
ckanext.dgu.controllers.publisher:PublisherController

Sets 'ckan.auth.profile' to 'publisher' (and same for harvesting: 'ckan.harvest.auth.profile' = 'publisher'). Publisher controller, based on Group:

• Publisher browse page includes hierarchy, alpha-browse and search.
• Apply to be a publisher editor or admin.
• Edit admins/editors for a publisher
• Publisher read shows publisher hierarchy, search pane with results and pager
• Display publisher's admins/editors for appropriate users.
• Report pages - users not assigned to groups, groups without admins, publishers vs users, users When user is created, flashes "You can now apply for publisher access"

dgu_theme

ckanext.dgu.plugin:ThemePlugin
ckanext.dgu.controllers.data:DataController
ckanext.dgu.controllers.tag:TagController
ckanext.dgu.controllers.reports:ReportsController
from ckanext.dgu.lib import helpers
ckanext/dgu/theme/templates
ckanext/dgu/theme/public

Data, Tag and Reports Controllers. Templates, helper functions. Random extras:

• Viewing user names (e.g. in History) change them for dept if not editor/admin.
• Add 'Vary: Cookie' header to all responses.

dgu_search

ckanext.dgu.plugin:SearchPlugin
from ckanext.dgu.search_indexing import SearchIndexing

Add fields to search index. Default sort-by. Escape SOLR characters. Search field weighting adjusted.

dgu_publisher_form

ckanext.dgu.forms.publisher_form:PublisherForm

New group form, type 'publisher' with schema. Added fields: contact, foi contacts, category, abbreviation.

dgu_dataset_form

ckanext.dgu.forms.dataset_form:DatasetForm
ckanext.dgu.forms.validators
ckanext.dgu.schema:GeoCoverageType

New dataset form. Lots of schema and validation customisation.

dgu_api

ckanext.dgu.plugin:ApiPlugin
ckanext.dgu.controllers.api:DguApiController
ckanext.dgu.controllers.api:DguReportsController
ckanext.dgu.lib.reports

Util API for Drupal - latest-datasets (front page), dataset-count (front page), revisions (unused). Reports, starting with organisation_resources.

dgu_resource_updates/dgu_resource_url_updates

ckanext.dgu.plugin:ResourceModificationPlugin ckanext.dgu.plugin:ResourceURLModificationPlugin

When a resource is created/deleted/URL-changed, this updates the last_major_modification date of its package.

Non-plugin code

ckanext/dgu/schema.py - mostly not used now ckanext/dgu/drupalclient.py - for getting user info from Drupal dgu/ckanext/dgu/bin/ - scripts used at one time or another dgu/ckanext/dgu/commands/ - scripts used at one time or another

Install

This is how to install ckanext-dgu, ckan and their dependencies into a python virtual environment::

virtualenv pyenv
pip -E pyenv install -e git+https://github.com:okfn/ckanext-dgu.git#egg=ckanext-dgu
pip -E pyenv install -e git+https://github.com/okfn/ckan.git#egg=ckan
pip -E pyenv install -r pyenv/src/ckan/pip-requirements.txt
pip -E pyenv install -r pyenv/src/ckanext-dgu/pip-requirements.txt

There are plenty of other little details about getting CKAN running under Apache, SOLR config etc in the CKAN README.

When configuring Apache, DGU enables some modules SSL and for ckanext-os. This is the complete list:: sudo a2enmod proxy wsgi headers proxy_http rewrite ssl

Configuration

Different parts of the DGU extension require options to be set in the CKAN configuration file (.ini) in the [app:main] section

To use the DGU package form specify::

ckan.plugins = dgu_form dgu_package_form

For the Drupal RPC connection (for user data etc.) supply the hostname, and credentials for HTTP Basic Authentication (if necessary)::

dgu.xmlrpc_domain = drupal.libre.gov.fr:80
dgu.xmlrpc_username = ckan
dgu.xmlrpc_password = letmein

The DGU-version of the SOLR schema is required instead of the CKAN SOLR schema. Whether you use a single or mult-core SOLR setup, you'll need a link to the DGU SOLR schema like this::

sudo ln -s /home/okfn/pyenv/src/ckanext-dgu/config/solr/schema-1.4-dgu.xml /etc/solr/conf/schema.xml

Initialise database

Creating the database is as usual for CKAN::

db=dgu
db_user=dgu
sudo -u postgres createuser -S -D -R -P $db_user
sudo -u postgres createdb -O $db_user $db

And because we usually use ckanext-dgu also with ckanext-spatial, then PostGIS needs setting up too::

sudo -u postgres psql -d $db -f /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql -v ON_ERROR_ROLLBACK=on
sudo -u postgres psql -d $db -f /usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql
sudo -u postgres psql -d $db -c "ALTER TABLE spatial_ref_sys OWNER TO $db_user;"
sudo -u postgres psql -d $db -c "ALTER TABLE geometry_columns OWNER TO $db_user;"

(For more info on that, see the README for ckanext-spatial.)

Usage

There is a front-page added to CKAN which describes the Catalogue APIs. The usual CKAN front-page has been moved to /ckan/ .

e.g.::

http://127.0.0.1:5000/ckan/

Scripts

There are a number of command-line scripts for processing data. Over the years they have been created in three different ways, but more recently we've been focussing on just creating stand-alone python scripts.

To run a python script (in the ckanext/dgu/bin directory) you activate the environment and then call it with python::

. pyenv/bin/activate
python ckanext/dgu/bin/dataset_change_publisher.py --help

To run a console script (listed in setup.py) you should activate the environment first. For example::

. pyenv/bin/activate
bulk_delete --help

To run a paster command (listed in setup.py) you activate the environment and if your CWD is not ckanext-dgu, specify it as the plugin::

. pyenv/bin/activate
paster --plugin=ckanext-dgu prod --help

Assets

The DGU theme uses assets (images, javascript, css) from this repo and the shared assets repo::

https://github.com/datagovuk/shared_dguk_assets

Both repos should be cloned next to each other on developer and server machines. If this is not possible then you need to set the dgu.shared_assets_timestamp_path config option to tell CKAN where the shared assets timestamp file is. e.g.::

dgu.shared_assets_timestamp_path = /vagrant/src/shared_dguk_assets/assets/timestamp

Assets are stored in the repo in 'source' form - the form easiest for developers to edit them in. Before they can be served, Grunt must be run on both repos to create the 'public' versions of these files. This does concatenation, minification, compilation of the less, and recording a timestamp (see Gruntfile.js for details).

Source:

images:     ckanext/dgu/theme/src/images
javascript: ckanext/dgu/theme/src/scripts
css (less): ckanext/dgu/theme/src/css

And when you run grunt, you get:

images:     ckanext/dgu/theme/public/images
javascript: ckanext/dgu/theme/public/scripts
css:        ckanext/dgu/theme/public/css
timestamp:  ckanext/dgu/theme/timestamp.py

Read more about Grunt installation and running it: https://github.com/datagovuk/shared_dguk_assets/blob/master/README.md

The shared assets need to be served at /assets. On a deployment server, setup nginx or apache to do this. A developer running under paster will find that the shared assets are served autoimatically, as long as the repo is cloned alongside ckanext-dgu and that this config option is not set: dgu.shared_assets_timestamp_path.

Tests

#####Unit and functional tests

To test the DGU extension you need the setup with CKAN (see above) and creation of a configured pyenv/src/ckan/development.ini (see http://docs.ckan.org/en/latest/install-from-source.html ).

To run the tests::

{pyenv}/bin/activate
cd {pyenv}/ckanext-dgu
nosetests --ckan ckanext/dgu/tests/

or run them from another directory by specifying the test.ini::

nosetests --ckan --with-pylons={pyenv}/src/ckanext-dgu/test.ini {pyenv}/src/ckanext-dgu/ckanext/dgu/tests/

You can either run the 'quick and dirty' tests with SQLite or more comprehensively with PostgreSQL. Set --with-pylons to point to the relevant configuration - either test.ini or test-core.ini (both from the ckanext-dgu repo, not the ckan one). For more information, see http://docs.ckan.org/en/latest/install-from-source.html .

#####Browser tests

Selenium is used to test a site is operating to a basic minimum standard, and specific checks on javascript elements.

To run the Selenium tests, TODO

Troubleshooting

#####Address and Connection errors

• socket.error: [Errno 98] Address already in use
• error: [Errno 111] Connection refused

These errors usually means a previous run of the tests has not cleaned up the Mock Drupal process. You can verify that::

$ ps a | grep mock_drupal
4748 pts/8    S      0:00 /home/dread/hgroot/pyenv-dgu/bin/python /home/dread/hgroot/pyenv-dgu/bin/paster --plugin=ckanext-dgu mock_drupal run -q

Now kill it before running the tests again::

$ kill 4748

#####Config errors

• DrupalXmlRpcSetupError: Drupal XMLRPC not configured.

The missing settings that result in this error are to be found in {pyenv}/src/ckanext-dgu/test-core.ini which is also imported into {pyenv}/src/ckanext-dgu/test.ini, so make sure you are specifying either of these config files in your nosetests --with-pylons parameter.

• ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ERROR: [f752f33380e3eec1379cfb89e0fdded8] multiple values encountered for non multiValued field parent_publishers: [london-borough-of-barnet, local-authorities]

This is due to SOLR using the CKAN SOLR schema, rather than the specific DGU one. Change it using the ln -s command above, followed by stopping and starting SOLR.

Documentation

DGU is an extension for CKAN: http://ckan.org

This README file is part of the DGU Developer Documentation, stored in the ckanext-dgu repo at ckanext-dgu/doc.

The Developer Docs can be built using Sphinx <http://sphinx.pocoo.org/>_::

  python setup.py build_sphinx