Political contributions mini search engine

A simple site to make political contribution data from the 総務省 more searchable.

Requirements

To run the site:

Flask
SQLAlchemy
Flask-Restless
Flask-SQLAlchemy
Flask-Babel

To do scraping:

requests
Poppler pdf utils
lxml

To make thumbnails:

optipng
ImageMagick

For running the OCR process:

Tesseract >=3.04 (+ eng and jpn lang packs)

Initialising

This repo contains no data. To fetch the data you need to:

Scrape PDFs from the 総務省 using tools/scrape.py
Initialise the DB with ./main.py initdb
Generate thumbnails for the PDFs with tools/make_thumbnails.py - this can take a long time as it runs optipng on each doc - use -n to go faster and make slightly larger PNGs.
Import the documents into the database with tools/importdocs.py - ideally run this 3 times: 1. groups only (-g), 2. defer enabled (no options), 3. no-defer (-n).

Each tool has additional options that can be applied. Run with --help to learn more.

License

The code in this repository is made available under the BSD license. See LICENSE.