Skip to content

Commit

Permalink
fix: Update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
rsavoye committed Oct 5, 2023
1 parent 3375b83 commit e33f28f
Show file tree
Hide file tree
Showing 76 changed files with 3,933 additions and 20 deletions.
64 changes: 62 additions & 2 deletions docs/about.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,63 @@
# osm-raw-data
# osm-rawdata

🕮 [Documentation](https://hotosm.github.io/osm-rawdata/)

These is a module to work with
[OpenStreetMap](https://www.openstreetmap.org) data using postgres and
a custom database schema. This code is derived from the [HOT Export
Tool](https://export.hotosm.org/en/v3/),
[osm-fieldwork](https://pypi.org/project/osm-fieldwork/), and
[Underpass](https://github.com/hotosm/underpass), and the [Raw Data
API](https://github.com/hotosm/raw-data-api), which is the new FastAPI
backend for the HOT Export Tool.

Since multiple projects need to do data extracts from OpenStreetMap in
a flexible way, this was designed to have a single body of code to
maintain.

## Installation

To install osm-rawdata, you can use pip. Here are two options:

- Directly from the main branch:
`pip install git+https://github.com/hotosm/osm-rawdata.git`

- Latest on PyPi:
`pip install osm-rawdata`

## The Database Schema

This project is heavily dependant on postgres and postgis. This schema
was optimized for data anaylsis more than display purposes. The
traditional schema use for OSM shows how it has evolved over the
years. Some tags are columns (usually empty), and others get put into
an **jsonb** *tag* column where they have to be accessed
directly. One big change in this datbase schema is all the tags are in
a single column, reducing the data size considerably, while also being
easier to query in a consistent manner. In the past a developer had to
keep track of what was a column, and what was in the *tags* column,
which was inefficient.

This schema has 4 tables, similar to the traditional ones. OSM data is
imported using [osm2pgsql](https://osm2pgsql.org/) but uses a
[lua](http://www.lua.org/) script to create the custom
schema. This module's usage is all read-only, as Underpass can keep
the raw data updated every minute, and we just want to access that
data.

Things get more interesting as this module supports both a local
database and a remote one. They use different query languages. To
simplify this, a configuration file is used, which then generates the
proper query syntax.

## The Config File

This reads in two different formats that describe the eventualy SQL
query. The YAML format was originally used by Export Tool, but later
abandoned for a JSON format. The YAML format was adopted by the
osm-fieldwork project before this transistion happened, so uses an
enhanced version to define the queries.

The JSON format is also supported, both parsing the config file and
also generating that query from a YAML config file.

This is a project for accessing an OSM database
10 changes: 10 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,13 @@ heading_level: 3
options:
show_source: false
heading_level: 3

::: osm_rawdata.geofabrik.GeoFabrik
options:
show_source: false
heading_level: 3

::: osm_rawdata.config.QueryConfig
options:
show_source: false
heading_level: 3
Binary file added docs/apidocs/html/bc_s.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/apidocs/html/bc_sd.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/apidocs/html/bdwn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
<map id="osm_rawdata.config.QueryConfig" name="osm_rawdata.config.QueryConfig">
<area shape="rect" id="node1" title=" " alt="" coords="5,131,201,325"/>
<area shape="rect" id="node2" title=" " alt="" coords="72,5,135,83"/>
</map>
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
efdbeba44ed4742460cc5531939adb38
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
<map id="osm_rawdata.config.QueryConfig" name="osm_rawdata.config.QueryConfig">
<area shape="rect" id="node1" title=" " alt="" coords="5,131,201,325"/>
<area shape="rect" id="node2" title=" " alt="" coords="72,5,135,83"/>
</map>
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
efdbeba44ed4742460cc5531939adb38
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
<map id="osm_rawdata.geofabrik.GeoFabrik" name="osm_rawdata.geofabrik.GeoFabrik">
<area shape="rect" id="node1" title=" " alt="" coords="5,131,212,252"/>
<area shape="rect" id="node2" title=" " alt="" coords="77,5,140,83"/>
</map>
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
497ae777a2a8c2381ca5fc93b9e1f858
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
<map id="osm_rawdata.geofabrik.GeoFabrik" name="osm_rawdata.geofabrik.GeoFabrik">
<area shape="rect" id="node1" title=" " alt="" coords="5,131,212,252"/>
<area shape="rect" id="node2" title=" " alt="" coords="77,5,140,83"/>
</map>
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
497ae777a2a8c2381ca5fc93b9e1f858
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/apidocs/html/closed.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/apidocs/html/doc.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/apidocs/html/docd.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit e33f28f

Please sign in to comment.