Skip to content

Latest commit

 

History

History
63 lines (45 loc) · 3.61 KB

README.md

File metadata and controls

63 lines (45 loc) · 3.61 KB

hansard project

Build Status

I'm just starting out with an analysis of the UK Parlimentary Archive. The broad goals at the minute are to create language models from the speaches given by UK officials in the House of Commons and the House of Lords. I have many interestng questions to answer about this dataset, please take a look at the issues here to see some that are being investigated.

Getting the data

There are 2 ways to get the data

This project contains scripts to access both: A small bash file in the data folder can be used to automate downloading of XML files, and a scrapy project for scraping data from the API. use the command scrapy crawl comments -o comments.json to extract data from the API.

Analysis

Data analysis is prototyped using ipython notebooks. You can view them using the notebook viewer:

  1. Data Wrangling Obtaining data from the API, cleaning it and converting it to a useful datastructure.
  2. Generating model representations of the data The raw text is cleaned of punctuation and tokenised. Bag of Words and IF/IDF models are created.
  3. PCA and clustering of Speakers - do labeled groups cluster using PCA? i.e do speakers from the house of commons / lords cluster together? Do speakers from the different poitical parties cluster together?
  4. LDA Visualization of one parlimentary sitting.

Related Projects

development

this project is using python virtual environments to manage the dependancies locally instead of installing packages globaly on your system. to setup a new virtual environmant use

# initialise your virtual environment
$ pyvenv venv
# activate the virtual environment
source ./venv/bin/activate
# install openblas
sudo apt-get install gfortran libopenblas-dev liblapack-dev
# install the project dependancies
pip install -r requirements.txt

If scikit-learn install fails there may be trouble running with python 3. Follow these instrunctions for Ubuntu:

sudo apt-get install build-essential python3-dev python3-setuptools python3-numpy python3-scipy python3-pip libatlas-dev libatlas3gf-base
sudo pip3 install scikit-learn
sudo update-alternatives --set libblas.so.3 \
    /usr/lib/atlas-base/atlas/libblas.so.3
sudo update-alternatives --set liblapack.so.3 \
    /usr/lib/atlas-base/atlas/liblapack.so.3

Contributers