Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

POSTGIS_BULK_INIT_ERRORS #3

Closed
Jean-Baptiste-Lasselle opened this issue Oct 21, 2018 · 12 comments
Closed

POSTGIS_BULK_INIT_ERRORS #3

Jean-Baptiste-Lasselle opened this issue Oct 21, 2018 · 12 comments

Comments

@Jean-Baptiste-Lasselle
Copy link
Owner

Jean-Baptiste-Lasselle commented Oct 21, 2018 •
edited
Loading

After computing Australia all night long, PostGIS ended up in a state with errors, and as I acces the Leaflet app through NGINX, I get a connection refused:

app is ruined

Here are the logs after a whole night computing the australia data :

[jibl@pc-100 carto-proto]$ docker logs -f carto-proto_web_1
^C
[jibl@pc-100 carto-proto]$ docker logs -f rendereurpoulet
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
^C
[jibl@pc-100 carto-proto]$ docker logs -f postgis
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
setting password ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
****************************************************
WARNING: No password has been set for the database.
         This will allow anyone with access to the
         Postgres port to access your database. In
         Docker's default configuration, this is
         effectively any other container on the same
         system.

         Use "-e POSTGRES_PASSWORD=password" to set
         it in "docker run".
****************************************************
waiting for server to start....LOG:  database system was shut down at 2018-10-21 03:12:09 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/postgis.sh
CREATE DATABASE
UPDATE 1
Loading PostGIS extensions into template_postgis
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
Loading PostGIS extensions into gis
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
osm2pgsql version 0.92.0 (64 bit id space)

Using built-in tag processing pipeline
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=11
Mid: pgsql, scale=100 cache=800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: /australia-oceania-latest.osm.pbf
Using PBF parser.
Processing: Node(84621k 152.5k/s) Way(1024k 1.63k/s) Relation(0 0.00/s)
Processing: Node(84621k 152.5k/s) Way(6112k 3.61k/s) Relation(104250 88.35/s)  parse time: 3429s
Node stats: total(84621444), max(6001147314) in 555s
Way stats: total(6112917), max(636363183) in 1693s
Relation stats: total(104252), max(8834677) in 1180s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline

Going over pending ways...
	3823949 ways are pending

Using 4 helper-processes
Finished processing 3823949 ways in 822 s

3823949 Pending ways took 822s at a rate of 4652.01/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending relations...
	0 relations are pending

Using 4 helper-processes
Finished processing 0 relations in 0 s

Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
Stopping table: planet_osm_nodes
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_ways
Building index on table: planet_osm_ways
Sorting data and creating indexes for planet_osm_point
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_roads
Stopped table: planet_osm_rels in 51s
Copying planet_osm_point to cluster by geometry finished
Creating geometry index on planet_osm_point
Copying planet_osm_roads to cluster by geometry finished
Creating geometry index on planet_osm_roads
Creating osm_id index on planet_osm_roads
Creating indexes on planet_osm_roads finished
All indexes on planet_osm_roads created in 298s
Completed planet_osm_roads
LOG:  checkpoints are occurring too frequently (23 seconds apart)
HINT:  Consider increasing the configuration parameter "max_wal_size".
Copying planet_osm_line to cluster by geometry finished
Creating geometry index on planet_osm_line
Copying planet_osm_polygon to cluster by geometry finished
Creating geometry index on planet_osm_polygon
Creating osm_id index on planet_osm_point
Creating indexes on planet_osm_point finished
All indexes on planet_osm_point created in 694s
Completed planet_osm_point
LOG:  checkpoints are occurring too frequently (29 seconds apart)
HINT:  Consider increasing the configuration parameter "max_wal_size".
LOG:  checkpoints are occurring too frequently (29 seconds apart)
HINT:  Consider increasing the configuration parameter "max_wal_size".
LOG:  checkpoints are occurring too frequently (29 seconds apart)
HINT:  Consider increasing the configuration parameter "max_wal_size".
Creating osm_id index on planet_osm_line
LOG:  checkpoints are occurring too frequently (26 seconds apart)
HINT:  Consider increasing the configuration parameter "max_wal_size".
Creating indexes on planet_osm_line finished
All indexes on planet_osm_line created in 859s
Completed planet_osm_line
Creating osm_id index on planet_osm_polygon
Creating indexes on planet_osm_polygon finished
All indexes on planet_osm_polygon created in 947s
Completed planet_osm_polygon
Stopped table: planet_osm_ways in 1903s
node cache: stored: 55343970(65.40%), storage efficiency: 52.78% (dense blocks: 1801, sparse nodes: 45051905), hit rate: 68.55%

Osm2pgsql took 6159s overall

LOG:  received fast shutdown request
LOG:  aborting any active transactions
LOG:  autovacuum launcher shutting down
waiting for server to shut down....LOG:  shutting down
...............LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

LOG:  database system was shut down at 2018-10-21 04:55:19 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
@Jean-Baptiste-Lasselle
Copy link
Owner Author

Jean-Baptiste-Lasselle commented Oct 21, 2018 •
edited
Loading

docker logs :

  • pour le conteneur nginx, j'ai un docker logs complètement vide après toute une nuit de compute
  • pour le conteneur renderer, j'ai les logs suivants (l'état est resté "en attente de la BDD " pdt toute la nuit, et l'st toujours, après la messe du Dimanche et les pâtisseries à la Boulangerie de l'église...) 👍
[jibl@srv-10 carto-proto]$ docker logs -f rendereurpoulet
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...

@Jean-Baptiste-Lasselle
Copy link
Owner Author

/assign @Jean-Baptiste-Lasselle 👍

  • try a much smaller *.PBF than all of australia
  • then cluster up the postgis so it becomes a beast
  • re-test with big australia
  • re-start all that cycle until I took over the entire world

@Jean-Baptiste-Lasselle
Copy link
Owner Author

Jean-Baptiste-Lasselle commented Oct 21, 2018 •
edited
Loading

Du nouveau :

J'ai déterminé que les couillons n'avaient pas défini la valeur de POSTGRES_USER dans le docker-compose.yml, si bien qu'aucun utilisateur n'était utilisable pour se connecter à la base de données qui elle était créée par la version couillon (cf. POSTGRES_DB dans le ./docker-compose.yml)

Désormais, ce processus osm2pgsql peut effectivement se connecter avec le user existant, et alors j'obitens un processus batch de téléchargement, de type ETL (extraction, après le téléchargement, il y aura traitement par osm2pgsql), qui lui, est donc bien long.
J'ai mesuré à la grosse fourchette, qu'au bout de 30 minutes, j'arrive à 20% de la tâche complétée, et avec des logs de la forme :

dowload pbf batch processing on ubuntu

20% <=> 30 minutes

100 % <=> 5 x 30 min = 2h30min (papa va regarder un film)

Ps Papa:

La forme des logs de l'exécutable osm2pgsql :

161950K .......... .......... .......... .......... .......... 26%  130K 2h32m
162000K .......... .......... .......... .......... .......... 26% 86.0K 2h32m
162050K .......... .......... .......... .......... .......... 26%  114K 2h32m
162100K .......... .......... .......... .......... .......... 26%  233K 2h32m
162150K .......... .......... .......... .......... .......... 26% 82.9K 2h32m
162200K .......... .......... .......... .......... .......... 26% 37.7K 2h32m

==>> Es-ce que le chiffre de la forme 2h32m , en fin de chaque ligne, ne serait pas l'évaluation dela durée de la tâche par wget? Non, c'est la durée restante de processing qui est ainsi affichée par osm2pgsql, car lorsque l'on consulte les logs en live d'un processus batch, il est courant que l'on ait aucune idée à priori, de l'heure à laquelle a commencé le processus batch.

Oui, typiqument UNIX, comme forme. ;)

Et en effet, après 30 minutes de film commencé 👍

durée restante logguée en effet décroissante

#whoyourdaddy

@Jean-Baptiste-Lasselle
Copy link
Owner Author

Jean-Baptiste-Lasselle commented Oct 21, 2018 •
edited
Loading

You know, daddy 's gona have to re-design it from scratch :

  • osm2pgsql is installed and excuted in the same container as the database server instance: that is stupid, importing new data in the run, wont be scalable though postgresql releases are definitely rock-solid, and fit for salability.
    #daddyplz put osm2pgsql inside a conainer of its own, so we can spin up a seperate declared scalable service for dockerc-compose && Kubernetes,

( bored daddy during adverts in middle of movie)

@Jean-Baptiste-Lasselle
Copy link
Owner Author

final logs of docker build process, including osm2pgsql successful execution 👍

  6100K .......... .......... .......... .......... .......... 89.6M
  6150K .......... .......... .......... .......... .......... 87.3M
  6200K .......... .......... .......... .......... .......... 89.1M
  6250K .......... .......... .......... .......... .......... 73.0M
  6300K .......... .......... .......... .......... .......... 91.7M
  6350K .......... .......... .......... .......... .......... 95.6M
  6400K .......... .......... .......... .......... .......... 77.2M
  6450K .......... .......... .......... .......... .......... 65.5M
  6500K .......... .......... .......... .......... .......... 58.3M
  6550K .......... .......... .......                          62.0M=1.0s

2018-10-21 20:48:02 (6.65 MB/s) - '1.075R-it.tar.gz' saved [6735736]

source-sans-pro-2.020R-ro-1.075R-it/
source-sans-pro-2.020R-ro-1.075R-it/EOT/
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Black.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-BlackIt.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Bold.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-BoldIt.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-ExtraLight.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-ExtraLightIt.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-It.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Light.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-LightIt.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Regular.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Semibold.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-SemiboldIt.eot
source-sans-pro-2.020R-ro-1.075R-it/LICENSE.txt
source-sans-pro-2.020R-ro-1.075R-it/OTF/
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Black.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-BlackIt.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Bold.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-BoldIt.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-ExtraLight.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-ExtraLightIt.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-It.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Light.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-LightIt.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Regular.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Semibold.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-SemiboldIt.otf
source-sans-pro-2.020R-ro-1.075R-it/README.md
source-sans-pro-2.020R-ro-1.075R-it/TTF/
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Black.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-BlackIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Bold.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-BoldIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-ExtraLight.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-ExtraLightIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-It.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Light.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-LightIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Regular.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Semibold.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-SemiboldIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/WOFF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Black.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-BlackIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Bold.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-BoldIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-ExtraLight.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-ExtraLightIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-It.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Light.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-LightIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Regular.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Semibold.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-SemiboldIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Black.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-BlackIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Bold.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-BoldIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-ExtraLight.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-ExtraLightIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-It.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Light.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-LightIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Regular.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Semibold.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-SemiboldIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Black.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-BlackIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Bold.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-BoldIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-ExtraLight.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-ExtraLightIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-It.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Light.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-LightIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Regular.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Semibold.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-SemiboldIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Black.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-BlackIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Bold.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-BoldIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-ExtraLight.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-ExtraLightIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-It.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Light.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-LightIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Regular.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Semibold.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-SemiboldIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/bower.json
source-sans-pro-2.020R-ro-1.075R-it/source-sans-pro.css
Removing intermediate container d911f04124da
 ---> 397fd637a568
Step 5/17 : RUN git clone https://github.com/dooman87/openstreetmap-carto.git
 ---> Running in d70e926736aa
Cloning into 'openstreetmap-carto'...
Removing intermediate container d70e926736aa
 ---> de4c0135bda1
Step 6/17 : WORKDIR /openstreetmap-carto
 ---> Running in 10593c68c94b
Removing intermediate container 10593c68c94b
 ---> a43571453916
Step 7/17 : ADD shapes/ data/
 ---> 79f5b61b1ccb
Step 8/17 : RUN shapeindex --shape_files     data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp     data/water-polygons-split-3857/water_polygons.shp     data/antarctica-icesheet-polygons-3857/icesheet_polygons.shp     data/antarctica-icesheet-outlines-3857/icesheet_outlines.shp     data/ne_110m_admin_0_boundary_lines_land/ne_110m_admin_0_boundary_lines_land.shp
 ---> Running in 882f8548394e
max tree depth:8
split ratio:0.55
processing data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp
9994
length=14943720
version=1000
type=5
extent:box2d(-20037508.3399999998509884,-14773407.4192944951355457,20037508.3399999998509884,20037508.3399999998509884)
 number shapes=5121
 number nodes=2468
processing data/water-polygons-split-3857/water_polygons.shp
9994
length=378892596
version=1000
type=5
extent:box2d(-20037508.3427892439067364,-14773407.4192944895476103,20037508.3427892439067364,20037508.3427892439067364)
 number shapes=42337
 number nodes=2131
processing data/antarctica-icesheet-polygons-3857/icesheet_polygons.shp
9994
length=34678596
version=1000
type=5
extent:box2d(-20037508.3427892178297043,-20037508.3427892550826073,20037507.2295943312346935,-8479237.0339447762817144)
 number shapes=14825
 number nodes=952
processing data/antarctica-icesheet-outlines-3857/icesheet_outlines.shp
9994
length=36341768
version=1000
type=3
extent:box2d(-20037508.3427892439067364,-20037508.3427892550826073,20037508.3427892439067364,-8479237.0339447762817144)
 number shapes=78810
 number nodes=1904
processing data/ne_110m_admin_0_boundary_lines_land/ne_110m_admin_0_boundary_lines_land.shp
9994
length=27922
version=1000
type=3
extent:box2d(-140.9977800000000059,-54.8968100000000021,141.0338517600138175,70.1641900000000049)
 number shapes=186
 number nodes=176
done!
Removing intermediate container 882f8548394e
 ---> 283d0715bfa8
Step 9/17 : RUN mkdir /map_data
 ---> Running in 0590b4cdbcfd
Removing intermediate container 0590b4cdbcfd
 ---> f90f0b430c72
Step 10/17 : VOLUME /map_data
 ---> Running in 1b7009b680df
Removing intermediate container 1b7009b680df
 ---> bcb35bb1f204
Step 11/17 : RUN mkdir /scripts
 ---> Running in 2b43ef0143ae
Removing intermediate container 2b43ef0143ae
 ---> 69601d9a89d7
Step 12/17 : COPY scripts/compile_style.sh /scripts
 ---> 233d0e36923d
Step 13/17 : COPY scripts/run_render.sh /scripts
 ---> 5933e5eb0c3a
Step 14/17 : COPY entrypoint.sh /
 ---> 4269f8ec06cb
Step 15/17 : RUN chmod 755 /scripts/*.sh &&     chmod 755 /entrypoint.sh
 ---> Running in ac496c54249d
Removing intermediate container ac496c54249d
 ---> 6c99491086df
Step 16/17 : WORKDIR /
 ---> Running in f210c3e8fc76
Removing intermediate container f210c3e8fc76
 ---> 8cf0112b049d
Step 17/17 : ENTRYPOINT /entrypoint.sh
 ---> Running in 9c983f5c6741
Removing intermediate container 9c983f5c6741
 ---> 85e1ac85aebd
Successfully built 85e1ac85aebd
Successfully tagged carto-proto_renderer:latest
WARNING: Image for service renderer was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Pulling web (nginx:1.11-alpine)...
1.11-alpine: Pulling from library/nginx
709515475419: Pull complete
4b21d71b440a: Pull complete
c92260fe6357: Pull complete
ed383a1b82df: Pull complete
Digest: sha256:5aadb68304a38a8e2719605e4e180413f390cd6647602bee9bdedd59753c3590
Status: Downloaded newer image for nginx:1.11-alpine
Creating postgis           ... done
Creating carto-proto_web_1 ... done
Creating rendereurpoulet   ... done
[jibl@pc-100 carto-proto]$

@Jean-Baptiste-Lasselle
Copy link
Owner Author

Jean-Baptiste-Lasselle commented Oct 21, 2018 •
edited
Loading

en tout moins de 2h30 c'est certain, pour le batch

Remarque 👍
selon le moment de la journée, le jour, l'import australie dure moins de 30 secondes au lieu de 2h30, et c'est le téléchargement du PBF, qui prend du temps, soit exactement l'instruction wget https://download.geofabrik.de/australia-oceania-latest.osm.pbf dans le Dockerfile postgis :

Step 4/8 : RUN git clone https://github.com/dooman87/openstreetmap-carto.git &&     wget https://download.geofabrik.de/australia-oceania-latest.osm.pbf
 ---> Running in 0557eb4cb2f1
Cloning into 'openstreetmap-carto'...
--2018-10-22 09:47:11--  https://download.geofabrik.de/australia-oceania-latest.osm.pbf
Resolving download.geofabrik.de (download.geofabrik.de)... 88.99.24.113, 88.99.105.98
Connecting to download.geofabrik.de (download.geofabrik.de)|88.99.24.113|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 633086720 (604M) [application/octet-stream]
Saving to: ‘australia-oceania-latest.osm.pbf’

     0K .......... .......... .......... .......... ..........  0%  592K 17m25s
    50K .......... .......... .......... .......... ..........  0%  420K 20m58s
   100K .......... .......... .......... .......... ..........  0%  654K 19m14s

Ce n'est donc pas le traitement osm2pgsql . Ilffaudra doc optimiser cela, avec un service batch scallable, chargé de télécahrger les PBF avec promesse tenue.

@Jean-Baptiste-Lasselle
Copy link
Owner Author

J'ai toujours des anomalies (les tuiles n'aparaissen tpas). Symptômes : le conteneur postgis indique dans ses logs un arrêt incorrect + conteneur rendereurpoulet indiquant qu'il est en attente de la BDD + la connexion à bdd par psql du conteneur rendereurpoulet vers le conteneur postgis.

[jibl@pc-100 carto-proto]$ docker logs -f rendereurpoulet
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
^C
[jibl@pc-100 carto-proto]$ docker logs -f postgis
LOG:  database system was interrupted; last known up at 2018-10-21 21:18:53 UTC
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  invalid record length at 0/2EE27D8
LOG:  redo is not required
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
^C
[jibl@pc-100 carto-proto]$ docker exec -it rendereurpoulet sh -c "apt-get update -y && apt-get install -y postgresql-client"
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:2 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]
Get:3 http://archive.ubuntu.com trusty Release.gpg [933 B]
Get:4 http://archive.ubuntu.com trusty Release [58.5 kB]
Get:5 http://archive.ubuntu.com trusty-updates/main Sources [524 kB]
Get:6 http://archive.ubuntu.com trusty-updates/restricted Sources [6449 B]
Get:7 http://archive.ubuntu.com trusty-updates/universe Sources [268 kB]
Get:8 http://archive.ubuntu.com trusty-updates/main amd64 Packages [1387 kB]
Get:9 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [21.4 kB]
Get:10 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [634 kB]
Get:11 http://archive.ubuntu.com trusty-security/main Sources [212 kB]
Get:12 http://archive.ubuntu.com trusty-security/restricted Sources [5050 B]
Get:13 http://archive.ubuntu.com trusty-security/universe Sources [104 kB]
Get:14 http://archive.ubuntu.com trusty-security/main amd64 Packages [967 kB]
Get:15 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [18.1 kB]
Get:16 http://archive.ubuntu.com trusty-security/universe amd64 Packages [338 kB]
Get:17 http://archive.ubuntu.com trusty/main Sources [1335 kB]
Get:18 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
Get:19 http://archive.ubuntu.com trusty/universe Sources [7926 kB]
Get:20 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:21 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:22 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Fetched 23.3 MB in 3s (6531 kB/s)                        
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libpq-dev libpq5 postgresql-client-9.3 postgresql-client-common
Suggested packages:
  postgresql-doc-9.3 postgresql-9.3
The following NEW packages will be installed:
  postgresql-client postgresql-client-9.3 postgresql-client-common
The following packages will be upgraded:
  libpq-dev libpq5
2 upgraded, 3 newly installed, 0 to remove and 174 not upgraded.
Need to get 1047 kB of archives.
After this operation, 3224 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpq-dev amd64 9.3.24-0ubuntu0.14.04 [140 kB]
Get:2 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpq5 amd64 9.3.24-0ubuntu0.14.04 [78.5 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ trusty-updates/main postgresql-client-common all 154ubuntu1.1 [25.4 kB]
Get:4 http://archive.ubuntu.com/ubuntu/ trusty-updates/main postgresql-client-9.3 amd64 9.3.24-0ubuntu0.14.04 [797 kB]
Get:5 http://archive.ubuntu.com/ubuntu/ trusty-updates/main postgresql-client all 9.3+154ubuntu1.1 [5048 B]
Fetched 1047 kB in 0s (7039 kB/s)
(Reading database ... 40968 files and directories currently installed.)
Preparing to unpack .../libpq-dev_9.3.24-0ubuntu0.14.04_amd64.deb ...
Unpacking libpq-dev (9.3.24-0ubuntu0.14.04) over (9.3.15-0ubuntu0.14.04) ...
Preparing to unpack .../libpq5_9.3.24-0ubuntu0.14.04_amd64.deb ...
Unpacking libpq5 (9.3.24-0ubuntu0.14.04) over (9.3.15-0ubuntu0.14.04) ...
Selecting previously unselected package postgresql-client-common.
Preparing to unpack .../postgresql-client-common_154ubuntu1.1_all.deb ...
Unpacking postgresql-client-common (154ubuntu1.1) ...
Selecting previously unselected package postgresql-client-9.3.
Preparing to unpack .../postgresql-client-9.3_9.3.24-0ubuntu0.14.04_amd64.deb ...
Unpacking postgresql-client-9.3 (9.3.24-0ubuntu0.14.04) ...
Selecting previously unselected package postgresql-client.
Preparing to unpack .../postgresql-client_9.3+154ubuntu1.1_all.deb ...
Unpacking postgresql-client (9.3+154ubuntu1.1) ...
Setting up libpq5 (9.3.24-0ubuntu0.14.04) ...
Setting up libpq-dev (9.3.24-0ubuntu0.14.04) ...
Setting up postgresql-client-common (154ubuntu1.1) ...
Setting up postgresql-client-9.3 (9.3.24-0ubuntu0.14.04) ...
update-alternatives: using /usr/share/postgresql/9.3/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up postgresql-client (9.3+154ubuntu1.1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
[jibl@pc-100 carto-proto]$ docker exec -it rendereurpoulet sh -c "psql -d gis -U renderer-user -h postgis -p 5432"
psql (9.3.24, server 9.5.14)
WARNING: psql major version 9.3, server major version 9.5.
         Some psql features might not work.
Type "help" for help.

gis=# 
gis-# \q
[jibl@pc-100 carto-proto]$

tests à papa

export APPEL_HTTP_TUILE_CHOPE_DS_FIREFOX=http://192.168.1.30:8080/14/14790/10054.png
docker exec -it carto-proto_web_1 sh -c "apk update -y && apk add curl && curl $APPEL_HTTP_TUILE_CHOPE_DS_FIREFOX"

Ma conclusion temporaire :
La BDD est nikel
Le conteneur NGINX est OK et l'appli SPA Leaflet OK aussi.

Dans le même temps, il ne reste que le conteneur du renderer qui lui indique dans ses logs, qu'il est en attente de la base de données.

ALors, je vais vérifier que la base de données existe, et ensuite, je me demanderai comment le renderer peut bien savoir quel est le nom de la BDD, ou le username/password PostGIS qu'il doit utiliser.

Mon avis :

  • D'abord j'ai de bonnes raisons de penser que notre poulet de chez "shinesolutions", a tout fait en local sur une seule et même machine, alller, je dirai sur une VM du genre virtualbox (le genre de technos de virtualisation qui est la plus rapidement accessible)
  • D'autre part, j'ai pu vérifier que la recette OPenStreetMAp fait usage de varibles d'environnements PostGIS, aka POSTGRES_DB et POSTGRES_USER
  • Enfin, je me souviens que les instructions d'openstreetmap concernent une installation de ce type : tout dans la même VM, zéro conteneur, zéro config réseau.

Donc, mon inbtuition :

  • Nos poulets de chez shinesolutions, ont fait un recopiage d'une bêtise monutmentale, d'un repo trouvé sur github, celui-ci de "dooman87"
  • Hormis le fait que "doom" est un jeu dont la seule motivation est la destruction totale, c'est un jeu tout à fait démodé, e complètement hors de son temps.

Si bien que je vais supposer qu'il suffit que je déclare des variables d'environnement POSTGRES_USER et POSTGRES_DB, pour le conteneur renbdereurpoulet dans le ./docker-compose.yml, pour que l'application renderer ait connaissance du nom de la BDD, et des credentials pour s'y connecter.

@Jean-Baptiste-Lasselle
Copy link
Owner Author

Jean-Baptiste-Lasselle commented Oct 22, 2018 •
edited
Loading

Pour plus tard 👍

Selon cette source :

Getting ready to load

osm2pgsql uses overcommit like many scientific and large data applications, which requires adjusting > a kernel setting to allow the data import to work successfully with multiple processes. Changing
overcommit is also recommended for renderd and postgresql.
sudo tee /etc/sysctl.d/60-overcommit.conf <<EOF
# Overcommit settings to allow faster osm2pgsql imports
vm.overcommit_memory=1
EOF
sudo sysctl -p /etc/sysctl.d/60-overcommit.conf

Une recommandation intéressante pour le RUN, selon la même source :

We also want to monitor the new database for size and other useful statistics
sudo munin-node-configure --sh | sudo sh
sudo service munin-node restart

résultat : eh ben ça a pas loupé nos poulerts de chez shinesolutions.com ont tellement rien branlé, qu'un import de 2h30 pour démarrer un dicker-compose, ça n,e les a pas du tout choqué, manifestement, et puis c'est comme si en cherchant 1 semaine, j'aivais trouvé de quoi réduire la même tâche à 30 secondes au lieu de 2h30, simplement en consultant la doc officielle OpenSTreetMap/PostGIS.

#jaimeraispasetreleurclient2h30putain

@Jean-Baptiste-Lasselle
Copy link
Owner Author

Oh! and there we are :

le commentaire sur une issue OpenStreetMap

@Jean-Baptiste-Lasselle
Copy link
Owner Author

Jean-Baptiste-Lasselle commented Oct 22, 2018 •
edited
Loading

Okay, tout va bien, sauf que le renderer n'arrive pas à accéder à la BDD POstGIS.
Et pour cause, nos poulets de chez shiesolutions, comme je l'avais soupçonné, ont bêtement recopié du code trouvé sur une page internet, dans des dockerfiles, et en plus, sans rien comprendre à docker.

En effet, si nos pouilets on un volume partagé entre le conteneur PostGIS, et le conteneur renderer, c'est parce que la recette originale d'OpenStreetMap, qu'ils ont trouvée [ici utilisée par dooman87], recommande d'utiliser un connexion par socket UNIX, pour les connexions du renderer vers la BDD PostGIS 👍

mais bien sûr, une connexion par unix socket....Sinon il y a RJ45, mais ça, c'était avant...

Pour que nous osyons bien d'accord, je cite le commentaire sur l'issue OpenStreetMap :

but when I omit the password, it can't connect to the database

Omit the user, password, hostname, and port, giving only the dbname. This will use local unix sockets to connect.
For extents use -20037508,-20037508,20037508,20037508. This is perhaps slightly sub-optimal for only poland, but will still work.
Also use 900913 for the SRS, not the custom string they gave.
Overall reading through https://www.mapbox.com/tilemill/docs/guides/postgis-work/ I see numerous problems with it.
My guide to setting up TileMill to work with OpenStreetMap Carto is https://hackpad.com/TileMill-I6rxRVszKMv but I haven't had time to finish it off.

Voilà pourquoi notre cerveau nous sert: analyser, avant de copier bêtelent, pour s'apercevoir après que l'on s'est appuyé sur un travail .... Améliorable.

No comment.

@Jean-Baptiste-Lasselle
Copy link
Owner Author

Jean-Baptiste-Lasselle commented Oct 22, 2018 •
edited
Loading

okay, so I have to understand how I can configure openstreetmap-carto 's connection to postgis

@Jean-Baptiste-Lasselle
Copy link
Owner Author

ok, about the long download of australia's PBF file, problem is solved by idempotence see. ./download.sh modifications, and "VAULT"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Jean-Baptiste-Lasselle