This repository is part of the Find Case Law project at The National Archives. For more information on the project, check the documentation.
A public interface to the service, allowing users to search archived items of case law.
This project uses Docker to create a consistent environment for local development.
On macOS and Windows, Docker requires Docker
Desktop to be installed. Linux
users should install the Docker engine using their distribution's package
manager or download a .deb or
.rpm
Once installed, we need to build our containers. We use
docker-compose to orchestrate the
building of the project's containers, one for each each service:
Our custom container responsible for running the application. Built from the official python 3.9 base image
The database service built from the official postgres image
NOTE: For any of the following commands to work, you must first install Fabric. (Homebrew is also an option: brew install fabric.) Once installed, you can type fab -l to see a list of available commands.
This app is intended to edit Judgments in the Marklogic database defined in ds-caselaw-marklogic.
Unless you are intending to do any database/Marklogic development work, it is simpler to access a
shared Marklogic database running on the staging environment than to build your own.
If you wish to run your own Marklogic instance, you will need to follow the setup instructions for it at ds-caselaw-marklogic.
The recommended alternative is to access the shared staging Marklogic database. The way you do this depends on where you work:
You will need to be using the dxw VPN. Retrieve the staging Marklogic credentials from dalmatian (or ask
one of the other developers/ops). Use these to fill MARKLOGIC_HOST, MARKLOGIC_USER and MARKLOGIC_PASSWORD
in your .env file (see step 2).
The MARKLOGIC_HOST should likely begin with caselaw
You will need VPN credentials from the dxw ops team, and the staging Marklogic credentials from one of the
dxw development team. Use these to fill MARKLOGIC_HOST, MARKLOGIC_USER and MARKLOGIC_PASSWORD
in your .env file (see step 2).
The MARKLOGIC_HOST should most likely begin with internal. Some people have experienced difficulties using
the domain name to connect to the server -- IP addresses may work better.
In both cases, when you run the application, you will be viewing data on staging Marklogic. This data is also used for testing and occasionally user research, so please exercise caution when creating/ editing content!
cp .env.example .envIf new environment variables are required, you might need to update .env to reflect that. Check .env.example for suitable default values
With the dependencies installed as described in Front end development
npm run buildfab buildYou might need to run this periodically if there are changes to the setup of the docker container; it's a good thing to run if your environment suddenly stops working.
If this fails early on, it's very likely that Docker isn't running, and you'll need to start it.
The docker-compose files in this repo and in ds-caselaw-marklogic both specify an external default network named caselaw.
We need to create a network with this name if it does not yet exist:
docker network create caselawNote If you are using the staging instance of Marklogic, you do not need to follow this step.
Switch to the location of ds-caselaw-marklogic and run:
docker-compose upAt this point you can run the following command to "quick start" the application:
fab runThis command takes care of the following:
- Starting all of the necessary Docker containers
- Installing any new python dependencies
- Applying any new database migrations
- Starting the Django development server
- Watching the Sass and JavaScript files for changes
You can then access the site in your browser:
(NOTE: The output of the asset and JS builds are logged in the ./assets.log file)
When starting up, if you encounter an error message like this:
ERROR: for postgres Cannot start service postgres: driver failed programming external connectivity on endpoint ds-caselaw-public-ui_postgres_1 (0fb7572d583761d3a348e8fd9139b0007638a17c6f91b15e8678f2575f94ffa7): Bind for 0.0.0.0:5432 failed: port is already allocated
It's because the editor UI project is still running, you'll need to reopen that project and run the command fab stop.
Now go back to the Public UI project and use the same command fab stop.
Now you can restart the project up again with fab run.
For day to day development, running fab run should provide you with all you need.
Other useful commands are:
Note that running this command will fail if you have already started the application with
fab run
fab startfab stopfab shRun the following inside the django container
python manage.py migratepython manage.py runserver_plus 0.0.0.0:3000Pytest unit tests can be run with:
fab testWe also have a suite of end to end tests (in the e2e_tests/ directory) written with playwright-pytest, which can be run with:
fab e2etestThese will run by default against the running django container. You can supply a baseURL argument to test against staging or production.
We use axe playwright to automatically check for accessibility issues on our pages. When adding a new page, you should also add a test to ensure it is accessible and stays accessible.
To add an automatic accessibility check you can add the assert_is_accessible check to your tests, for example:
def test_my_page(page: Page):
page.goto("/my_page")
assert_is_accessible(page)If the page is accessible, the console won't output anything other than the usual test output. If there are accessibility issues, there will be output that explains what the issues are and also helpful links explaining how to fix them.
fab coverageThis will generate an HTML file at htmlcov/index.html to view code coverage
We used the WeasyPrint library to handle PDF generation on the backend, but moved to producing PDFs via LibreOffice. But WeasyPrint is still used as a fallback if the PDF is not found in the S3 bucket.
If you are using the provided Docker images, the dependencies for WeasyPrint are bundled in the Dockerfile.
If you see an error containing this message:
If you are running the application locally, without using Docker (for example in a local virtualenv), then you need to follow the steps outlined here to install the WeasyPrint dependencies on your local machine. You may need to recreate your virtualenv after following these steps.
cannot load library 'gobject-2.0-0'Then it means the dependencies for WeasyPrint have not been installed correctly. Try rebuilding the docker image using the command docker-compose build django and then running fab run.
To use this, you will need to install pre-commit on your development machine, typically using pip install pre-commit. If you prefer Homebrew, you can use brew install pre-commit.
Install the git hooks configured in .pre-commit-config.yaml with:
pre-commit install
This will set up various checks including Python linting and style checks when you commit and push to the repo and alert you to any linting issues that will cause CI to fail.
Any commit that's merged to main needs to be signed, to ensure the identity of the author is who they say they are.
We recommend signing with your ssh key, as it's probably the easiest method of doing so. Assuming you already have an ssh key created, just follow the following steps:
- Add your SSH key as a signing key in your github account - note this is different to an authentication key, which you likely already have set up. You can use the same key for both purposes, but you need to add it separately for each one twice.
- In your terminal, run the following commands. This assumes you want to set up commit signing by default for all repositories. If you don't want this for whatever reason, leave out the
--globalflag (but in that case you'll have to remember to repeat these steps in every TNA repository you work on):- Enable signing with
git config --global commit.gpgsign true - Specify that we'll use SSH for signing with:
git config --global gpg.format ssh - Specify the key you'll use to sign. If it's not id_rsa.pub, give the correct path here:
git config --global user.signingkey ~/.ssh/id_rsa.pub
- Enable signing with
If you have already made some unsigned commits on a branch before setting up signing, you'll need to sign them before they can be merged. You can do this by rebasing, typically using for example git rebase --force-rebase main then doing a force push. Care should obviously be taken here however, especially if there's anyone else working on your branch!
- Ensure you have NodeJS & NPM installed.
Install these with:
npm i- Webpack and Babel for transpiling JavaScript
- Sass for compiling CSS
To use a local development copy of nationalarchives/ds-caselaw-frontend, for example to use your local copy of the shared CSS instead of what's currently in its github repo's main branch:
npm link ../path/to/your/copy/nationalarchives/ds-caselaw-frontendTo stop using your local copy, you can then run:
npm unlink ../path/to/your/copy/nationalarchives/ds-caselaw-frontend- To watch and build the site SASS, run
npm run start-sass - To modify styles, navigate to the
sassfolder in your editor.
The Judgment display CSS _judgment_text.scss should be the same between both
this application and ds-caselaw-editor-ui. Ensuring edits to this repository were being
replicated to the editor repository was tricky as it relied on the developers
remembering to make changes in both places.
Instead, we share the judgment CSS between both apps. This repository is the
"source of truth". Any edits made in ds-caselaw-public-ui which are then merged to main and included in a
production release, will be reflected in ds-caselaw-editor-ui (note that the changes have to be included in
a release before they are used in the editor).
Note that if a release is made on ds-caselaw-public-ui which contains edits to this CSS, a deployment will need to be
made to ds-caselaw-editor-ui to force that app to pick up the new version of the CSS. A deployment can be made via
dalmatian, without needing to do a full release of the editor.
_judgment_text.scss only contains styles for the HTML judgment view. Other CSS styles for the public UI and editor
UI applications are not shared.
In a new terminal session run npm run start-scripts to kick off a Webpack watch task
django commands need to be run within the django docker container, not on your machine itself, so from your terminal, you will need to first run fab sh, which will give you a console where you can run commands within the container (you'll see your terminal change from saying something like tim@Tims-Macbook at the start of each line to root@abcde12345). You can then run the commands you need to (such as python manage.py shell_plus), and when you're done, type the command exit to exit back out to your own machine again (the start of each line will change back).
Anyone with access to the Github repo can add or remove stop words from the search.
- Edit
judgments/fixtures/stop_words.pyto add or remove stop words as desired. Pay attention to the file format - the stop words must be in a Python list, quoted and followed by a comma. - Commit your changes to the repo and open a pull request as normal.
- Once the PR is merged, the modified list of stop words will be used in the search.
If there are new or changed environment variables, a dxw-er will need to run:
dalmatian service set-environment-variable -i caselaw-stg -e staging -s public -k NEW_ENV_VAR -v VALUE
(possibly with caselaw and prod for production and editor for the editor ui)
The main branch is automatically deployed with each commit. The deployed app can be viewed at https://staging.caselaw.nationalarchives.gov.uk/
To deploy to production:
- Create a new release.
- Set the tag and release name to
vX.Y.Z, following semantic versioning. - Publish the release.
- Automated workflow will then force-push that release to the
productionbranch, which will then be deployed to the production environment. - If you need to roll back to an earlier version, force-push that version to
productionmanually. Explictly rundalmatian service deploy(with the right-i,-e,-sflags) if it is not a new commit.
The production app is at https://caselaw.nationalarchives.gov.uk/
- Remember to
git pullthe freshest files - If
fab buildfails early, check that Docker is running? (Click the blue whale) - If the public-ui takes forever to load, check that the VPN is running -- you might need to change the IP address if DNS isn't working for you.
- If it was working, and you
git pulled, and now it isn't, re-runfab build. - If it's saying environment variables aren't set (especially after a
git pull), you might need to add lines to.env-- take them from.env.example