**********
Note: As part of opendataday.org, this codebase has recently been restructured
in order to make it more of a component (packaging coming soon), to increase
extensibility and hopefully encourage reuse. However, it's possible these
changes have introduced bugs (though hopefully not!) - if you have issues
installing or running, please do get in touch at <[email protected]>.
**********

MapIt
=====

The mapit directory is a standard GeoDjango app, and the project directory is
an example GeoDjango project using that app. Other top-level things are mostly
to fit within mySociety's standard deployment, or country specific data,
one-off scripts and so on.

MapIt has been installed on Debian lenny and squeeze, and on Ubuntu from 10.04
onwards. If GeoDjango and PostGIS run on it, it should theoretically be okay;
do let us know of any problems. Commands are intended to be run via the
terminal or over ssh.

Installation
------------

MapIt currently uses Postgres/PostGIS as its database backend - there's no reason 
why e.g. SpatiaLite could not be used successfully, but it has never been tried.

To install GeoDjango and PostGIS, please follow all the standard instructions
(including creating the template) at:
    http://docs.djangoproject.com/en/dev/ref/contrib/gis/install/#ubuntudebian
And anything else you need to set up Django as normal.

[ Note for UK/Ireland: Not only is the PostGIS that is installed missing SRID
900913, as the GeoDjango docs tell you, but both SRID 27700 (British National
Grid) and SRID 29902 (Irish National Grid) can be incorrect (and they're quite
important for this application!). After you've installed and got a PostGIS
template, log in to it and update the proj4text column of SRID 27700 to include
+datum=OSGB36, and update SRID 29902 to have +datum=ire65. This may not be
necessary, depending on your version of PostGIS, but do check. ]

You will also need a couple of other Debian packages, so install them:

    sudo apt-get install python-yaml memcached python-memcache git-core

You will also need to be using South to manage schema migrations.

Installation as a Django app
----------------------------

As mapit is a Django app, if you are making a GeoDjango project you can simply
use mapit from within your project like any normal Django app:
* Make sure it is on sys.path (a packaged install e.g. with 'pip install
  django-mapit' does this for you);
* Add 'mapit' and 'django.contrib.gis' to your INSTALLED_APPS;
* Add the following settings to settings.py:
  MAPIT_AREA_SRID - the SRID of your area data (if in doubt, set to 4326)
  MAPIT_COUNTRY - used to supply country specific functions (such as postcode
    validation). If you look at the country files in mapit/countries/ you can
    see how to add specialised country-specific functions.
  MAPIT_RATE_LIMIT - a list of IP addresses or User Agents excluded from rate limiting
* Set up a path in your main urls.py to point at mapit.urls.
* run './manage.py syncdb' and './manage.py migrate' to ensure the db is set up

This approach is new, so please let us know if something goes wrong or could be
improved.

Installation standalone with the example project
------------------------------------------------

A standard git clone will get you the repository:

    git clone git://github.com/mysociety/mapit.git

Now in a terminal, navigate to the mapit directory you've just cloned.

Copy conf/general.yml-example to conf/general.yml and edit it to point to your
local postgresql database, and edit the other settings as per the documentation
given in that file. If you don't know what SRID to use, delete that line or set
it to 4326. COUNTRY is used for e.g. differing sorts of postcode (zipcode)
validation - if you look at the country files in mapit/countries/ you can see
how to add specialised country-specific functions to validate postcodes etc.

If you're going to be importing big datasets, make sure that DEBUG is False;
otherwise, you'll run out of memory as it tries to remember all the SQL queries made.

Optionally, also turn off escape_string_warning in Postgres' config (unless you
want your server to run out of disc space logging all the errors, as ours did
the first time I imported overnight and didn't realise).

At this stage, you should be able to set up the database and run the
development server. Do add an admin user when prompted:

    cd project
    ./manage.py syncdb
    ./manage.py migrate mapit
    ./manage.py runserver

(Alternatively, set up a live web server however you wish - see the Deployment
Django documentation for details beyond the scope of this document.)

You can then visit http://localhost:8000/ and hopefully see the default MapIt homepage.
http://localhost:8000/admin/ should show the admin interface.

This is enough to have a working site. You can create areas and postcodes using the
admin interface and they will automatically work in the API-based front end.

However, if you have some bulk data you wish to import (which will make things
easier), you will need to import this data into MapIt. Below are some
instructions for such bulk import for the United Kingdom and Norway, which
should show the general format for how it's done.

United Kingdom
--------------

Here are the basic instructions to install OS OpenData and NSPD Open:

1. AREA_SRID in conf/general.yml should be 27700 (as Boundary-Line shapes are in OSGB).   
2. Download the latest Code-Point Open, Boundary-Line and NSPD from
   <http://parlvid.mysociety.org:81/os/>, and save/unzip in the data directory.
3. Change to the project directory, and create database tables:
   ./manage.py syncdb
   ./manage.py migrate mapit
4. Run the following in order:
   ./manage.py generation_create --commit --desc "Initial import."
   ./manage.py loaddata uk
   ./manage.py import_boundary_line --control=mapit.controls.first-gss --commit `ls ../../data/Boundary-Line/*.shp|grep -v high_water`
   # (You can run without --commit to do a dry run.)
   # first-gss in the above assumes the Boundary Line you're importing is
   # October 2010 or later, and uses the new GSS codes.
   ./manage.py find_parents
   ./manage.py import_codepoint ../../data/Code-Point-Open/*.csv
   ./manage.py scilly ../../data/Code-Point-Open/tr.csv
   ./manage.py import_nspd_ni_areas
   ./manage.py import_nspd_ni ../../data/ONSPD.csv
   ./manage.py import_nspd_crown_dependencies ../../data/ONSPD.csv
   ./manage.py generation_activate --commit

For notes on what was done to create generations as you can see on
mapit.mysociety.org, see the end of this file.

Notes on future releases
------------------------

When a new Code-Point is released, you should just be able to run import_codepoint 
and scilly; when new NSPD is out, import_nspd_ni if it's only postcodes that 
have changed, or import_nspd_ni_areas first if boundary changes too (this is 
incomplete, it doesn't use a control file like import_boundary_line does); 
when new Boundary-Line, import_boundary_line and find_parents.

In May 2011, the Northern Ireland Assembly boundaries move to match the current
Parliamentary boundaries - import_nspd_ni_areas needs changing to cope with that,
it currently only creates the current (pre May 2011) boundaries.

You can manually increase the generation_high_id when something is new and
something else isn't (most commonly, a new Boundary-Line means a new generation
for Great Britain, and you can then increase the Northern Ireland boundaries
manually to be in the new generation).

Norway
------

Here are the basic instructions to install data from OSM:

1. Set AREA_SRID in conf/general.yml to 4326 (as OSM shapes are in WGS84).  
2. cd bin and run "python osm_to_kml" to fetch OSM XML and convert it to KML.
3. Change to the project directory, and create database tables:
   ./manage.py syncdb
   ./manage.py migrate mapit
4. Run the following (you can run anything without --commit to do a dry run):
   ./manage.py generation_create --commit --desc "Initial import."
   ./manage.py loaddata norway
   ./manage.py import_norway_osm --commit ../../data/cache/*.kml
   ./manage.py import_area_unions --commit data/norway/regions.csv
   ./manage.py generation_activate --commit

Please see below for information on where osm_to_kml gets its OSM data from.

Alternatively, here are the basic instructions to install the N5000 data:

1. Set AREA_SRID in conf/general.yml to 4326 (as we'll put N5000 shapes into WGS84).  
2. Download N5000 from
   <http://www.statkart.no/nor/Land/Kart_og_produkter/N5000_-_gratis_oversiktskart/>
   and save/unzip in the data directory.
3. Change to the project directory, and create database tables:
   ./manage.py syncdb
   ./manage.py migrate mapit
4. Run the following (you can run anything without --commit to do a dry run):
   ./manage.py generation_create --commit --desc "Initial import."
   ./manage.py loaddata norway
   ./manage.py import_norway_n5000 --commit ../../data/N5000\ shape/N5000_AdministrativFlate.shp
   ./manage.py import_area_unions --commit data/norway/regions.csv
   ./manage.py generation_activate --commit

You should now be able to go to /point/4326/10.756389,59.949444 and have
Oslo returned as a result.

Norway OSM data
---------------

The way osm_to_kml works is to fetch a number of fixed and pre-defined
relation IDs from OSM - one (412437) containing all fylke, and then
one for each fylke containing all the kommune inside. These relations
should stay and (now they're correct) not change within OpenStreetmap,
though of course the underlying relations can have their boundaries
improved and so on. See the relation_ids list in the source of
bin/osm_to_kml if you'd like to see the other relation IDs.

The OSM tags 'name', and 'name:no' if 'name' is not set, are used to find
the primary name of the fylke and kommune. In addition, the values of
the tags 'name:smi', 'name:fi' are imported into mapit. Only the
primary name is shown in the mapit web pages and JSON data, while the
other names are stored in the database.

The kommune and fylke number (ID) is fetched from a the tag 'ref' in
OSM, and if it is missing a static list of such IDS in
mapit/data/norway/ids.csv is consulted using the name (for fylke) or
name+fylke (for kommune) as the key.

Improvements / patches
----------------------

Are welcome :)

ATB,
Matthew

=====================================

Notes on creating what's live
-----------------------------

When creating what you see at mapit.mysociety.org, to enable it to have
pre-2010 election boundaries, I ran the above (or rather, what existed at the
time, which is not identical) twice, once with 2009-10 Boundary-Line and then
the 2010-05 edition. I had to write the 2010-05 control file you can see, did
not re-run import_codepoint (as no postcodes had changed), and only ran the NI
stuff the second generation (as we only had current data). The commands I
basically ran are below.

Even worse, as I had to maintain IDs between our old and new versions of mapit,
I then matched up all IDs and names using the scripts in bin, manually inserted
some generation 10 areas (in data) for FixMyStreet and some generation 12 NI
WMC areas for WriteToThem, and manually added our test/fake areas that used to
be in code but can now happily sit in the database along with everything else.
You probably don't need any of that for your own install.

# Create inactive generation.
./manage.py import_boundary_line --control=mapit.controls.2009-10 `ls ../../data/Boundary-Line/2009-10/*.shp|grep -v high_water`
./manage.py import_codepoint ../../data/Code-Point-Open-2010-05/*.csv
./manage.py find_parents
# Not importing NI here, as that only has the current boundaries.
./manage.py scilly ../../data/Code-Point-Open-2010-05/tr.csv
# Make generation active, add another inactive generation
./manage.py import_boundary_line --control=mapit.controls.2010-05 `ls ../../data/Boundary-Line/2010-05/*.shp|grep -v high_water`
# import_codepoint not needed as it's the same and there's no P-in-P tests!
./manage.py find_parents
./manage.py scilly ../../data/Code-Point-Open-2010-05/tr.csv # I doubt the boundaries change! But updates the generation.
./manage.py import_nspd ../../data/NSPD-2010-05.csv # This is now split into two scripts, see below.
# Make generation active.