-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
76 changed files
with
3,933 additions
and
20 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,63 @@ | ||
# osm-raw-data | ||
# osm-rawdata | ||
|
||
🕮 [Documentation](https://hotosm.github.io/osm-rawdata/) | ||
|
||
These is a module to work with | ||
[OpenStreetMap](https://www.openstreetmap.org) data using postgres and | ||
a custom database schema. This code is derived from the [HOT Export | ||
Tool](https://export.hotosm.org/en/v3/), | ||
[osm-fieldwork](https://pypi.org/project/osm-fieldwork/), and | ||
[Underpass](https://github.com/hotosm/underpass), and the [Raw Data | ||
API](https://github.com/hotosm/raw-data-api), which is the new FastAPI | ||
backend for the HOT Export Tool. | ||
|
||
Since multiple projects need to do data extracts from OpenStreetMap in | ||
a flexible way, this was designed to have a single body of code to | ||
maintain. | ||
|
||
## Installation | ||
|
||
To install osm-rawdata, you can use pip. Here are two options: | ||
|
||
- Directly from the main branch: | ||
`pip install git+https://github.com/hotosm/osm-rawdata.git` | ||
|
||
- Latest on PyPi: | ||
`pip install osm-rawdata` | ||
|
||
## The Database Schema | ||
|
||
This project is heavily dependant on postgres and postgis. This schema | ||
was optimized for data anaylsis more than display purposes. The | ||
traditional schema use for OSM shows how it has evolved over the | ||
years. Some tags are columns (usually empty), and others get put into | ||
an **jsonb** *tag* column where they have to be accessed | ||
directly. One big change in this datbase schema is all the tags are in | ||
a single column, reducing the data size considerably, while also being | ||
easier to query in a consistent manner. In the past a developer had to | ||
keep track of what was a column, and what was in the *tags* column, | ||
which was inefficient. | ||
|
||
This schema has 4 tables, similar to the traditional ones. OSM data is | ||
imported using [osm2pgsql](https://osm2pgsql.org/) but uses a | ||
[lua](http://www.lua.org/) script to create the custom | ||
schema. This module's usage is all read-only, as Underpass can keep | ||
the raw data updated every minute, and we just want to access that | ||
data. | ||
|
||
Things get more interesting as this module supports both a local | ||
database and a remote one. They use different query languages. To | ||
simplify this, a configuration file is used, which then generates the | ||
proper query syntax. | ||
|
||
## The Config File | ||
|
||
This reads in two different formats that describe the eventualy SQL | ||
query. The YAML format was originally used by Export Tool, but later | ||
abandoned for a JSON format. The YAML format was adopted by the | ||
osm-fieldwork project before this transistion happened, so uses an | ||
enhanced version to define the queries. | ||
|
||
The JSON format is also supported, both parsing the config file and | ||
also generating that query from a YAML config file. | ||
|
||
This is a project for accessing an OSM database |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions
4
docs/apidocs/html/classosm__rawdata_1_1config_1_1QueryConfig__coll__graph.map
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
<map id="osm_rawdata.config.QueryConfig" name="osm_rawdata.config.QueryConfig"> | ||
<area shape="rect" id="node1" title=" " alt="" coords="5,131,201,325"/> | ||
<area shape="rect" id="node2" title=" " alt="" coords="72,5,135,83"/> | ||
</map> |
1 change: 1 addition & 0 deletions
1
docs/apidocs/html/classosm__rawdata_1_1config_1_1QueryConfig__coll__graph.md5
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
efdbeba44ed4742460cc5531939adb38 |
Binary file added
BIN
+15.6 KB
docs/apidocs/html/classosm__rawdata_1_1config_1_1QueryConfig__coll__graph.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions
4
docs/apidocs/html/classosm__rawdata_1_1config_1_1QueryConfig__inherit__graph.map
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
<map id="osm_rawdata.config.QueryConfig" name="osm_rawdata.config.QueryConfig"> | ||
<area shape="rect" id="node1" title=" " alt="" coords="5,131,201,325"/> | ||
<area shape="rect" id="node2" title=" " alt="" coords="72,5,135,83"/> | ||
</map> |
1 change: 1 addition & 0 deletions
1
docs/apidocs/html/classosm__rawdata_1_1config_1_1QueryConfig__inherit__graph.md5
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
efdbeba44ed4742460cc5531939adb38 |
Binary file added
BIN
+15.6 KB
docs/apidocs/html/classosm__rawdata_1_1config_1_1QueryConfig__inherit__graph.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions
4
docs/apidocs/html/classosm__rawdata_1_1geofabrik_1_1GeoFabrik__coll__graph.map
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
<map id="osm_rawdata.geofabrik.GeoFabrik" name="osm_rawdata.geofabrik.GeoFabrik"> | ||
<area shape="rect" id="node1" title=" " alt="" coords="5,131,212,252"/> | ||
<area shape="rect" id="node2" title=" " alt="" coords="77,5,140,83"/> | ||
</map> |
1 change: 1 addition & 0 deletions
1
docs/apidocs/html/classosm__rawdata_1_1geofabrik_1_1GeoFabrik__coll__graph.md5
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
497ae777a2a8c2381ca5fc93b9e1f858 |
Binary file added
BIN
+9.59 KB
docs/apidocs/html/classosm__rawdata_1_1geofabrik_1_1GeoFabrik__coll__graph.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions
4
docs/apidocs/html/classosm__rawdata_1_1geofabrik_1_1GeoFabrik__inherit__graph.map
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
<map id="osm_rawdata.geofabrik.GeoFabrik" name="osm_rawdata.geofabrik.GeoFabrik"> | ||
<area shape="rect" id="node1" title=" " alt="" coords="5,131,212,252"/> | ||
<area shape="rect" id="node2" title=" " alt="" coords="77,5,140,83"/> | ||
</map> |
1 change: 1 addition & 0 deletions
1
docs/apidocs/html/classosm__rawdata_1_1geofabrik_1_1GeoFabrik__inherit__graph.md5
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
497ae777a2a8c2381ca5fc93b9e1f858 |
Binary file added
BIN
+9.59 KB
docs/apidocs/html/classosm__rawdata_1_1geofabrik_1_1GeoFabrik__inherit__graph.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.