Showing natural areas on z5+

[kocio-pl]

Related to #2688
Implements parts of https://github.com/gravitystorm/openstreetmap-carto/blob/master/USECASES.md

Changes proposed in this pull request:

• Start rendering natural and semi-natural areas from z5 instead of z8+. Since we have usecases for some low zoom levels, it's time to show them there. z0-z4 might need separate attention, so it's good to make gradual progress and gather some experiences on z5-z7 first. This is also good opportunity for showing glacier and icesheets in a more balanced way (effectively reverting Render glaciers and icesheets from z8 instead of z6 #2970).

Examples (click on images to see the full size):

Poland

z5

z6

z7

Egypt

z5

z6

z7

[kocio-pl]

California, USA

z5

z6

z7

[jeisenbe]

Kocio-pl, 1. do you want to include wetland_mangrove, wetland_marsh etc? I didn’t see them in the changes. Mangroves are particularly large areas in the tropics. 2. Why isn’t the farmland showing up in the center of California? And along the Nile in Wgypt Is the waypixels limit too small? Lots of the farms there are 1 square mile (2.2 square kilometer) so its strange that they don’t show up. 3. Should we show natural parks and nature reserves too? In the USA the land in national forests and national parks is not usually tagged with landuse=forest -Joseph

…

On Thu, Oct 18, 2018 at 7:21 AM kocio-pl ***@***.***> wrote: *California, USA* z5 [image: nmxpmkaz] <https://user-images.githubusercontent.com/5439713/47119545-8933b100-d26b-11e8-993f-d65f7e1ae5a0.png> z6 [image: 4j2eavvj] <https://user-images.githubusercontent.com/5439713/47119548-8c2ea180-d26b-11e8-9617-00f61dbc5234.png> z7 [image: qpxu yzy] <https://user-images.githubusercontent.com/5439713/47119554-8f299200-d26b-11e8-9e21-f1b10ca018b4.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3458 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AoxshCNigGhAHM2c9Olr0hTqIwpNvV5_ks5ul61agaJpZM4Xo4vW> .

[kocio-pl]

Nepal

z5

z6

z7

[jeisenbe]

For comparison, here's what the French style looks like at z5, z6 and z7 in Poland. It looks like they manage to catch more landuse, perhaps by not filtering out some of the smaller polygons?

(There are other differences on the French style; eg darker colors, more towns, the gray borders and browner farmland colors, but hopefully we can still see the landuse differences.)

---

z5 Openstreetmap-cart with this PR

z5 French style

---

z6 This PR

z6 French

---

z7 This PR

z7 French

---

Edit: fixed order of images in z7

[kocio-pl]

Thanks for your reaction! I will try to look and answer all your questions, it's very helpful for me to discuss specific problems.

1. do you want to include wetland_mangrove, wetland_marsh etc? I didn’t see them in the changes.

I have tuned two more of them, but this is just for showing their background. As far as I can tell looking at the code, their particular symbols are rendered on z14+, and generic wetland pattern is already being rendered on z5+:

[int_wetland != null][zoom >= 5] {
    polygon-pattern-file: url('symbols/wetland.png');
    polygon-pattern-alignment: global;
}

Mangroves are particularly large areas in the tropics.

I have a problem with finding biggest areas of a given type in some area, which would be very handy for testing low zoom features. Maybe someone familiar with SQL could paste here some script I could use? On low zoom there is much harder to find visible examples than on medium and high zoom levels, simply because typically there are only few of them.

For example mangroves look like being very common in Cuba:

but this tool is not suitable for tuning low zoom rendering. There might be just a lot of small ones, since even on z7 there are no big wetland/mangrove areas except this swamp (click to see the full size image):

[jeisenbe]

I added some big mangroves in southwestern New Guinea about 1 or 2 months ago at 137° 30' 34.5"E, 5° 04' 04.9"S, the biggest is about 20km x 20km. But perhaps all the coastline and rivers will make it hard to see at z7? If all the wetland areas are rendering the same, then it's fine.

[kocio-pl]

z7 is for showing very big area. For example these "big" mangroves near Timika you mentioned are hardly visible in the middle and there are only some small forest patches around:

[jeisenbe]

Comparison of California in this PR (copied from above) and the French style.
The French style shows national parks and national forests starting at z7, and captures many smaller areas of farmland. It also shows areas in the desert that are not showing in this PR.

(It's also showing admin_level 6 boundaries, villages, and smaller roads earlier, while deemphasizing motorways.)

---

CA z5 This PR:

CA z5 French style:

---

CA z7 this PR: (copied and cropped)

CA z7 French:

---

[kocio-pl]

3. Should we show natural parks and nature reserves too? In the USA the land in national forests and national parks is not usually tagged with landuse=forest

We have z8+ for that. Natural parks and nature reserves are virtual borders for some kind of nature area, but this can be very different, not necessarily forest or only on some part of that (like here). They have a green shadow inside just as a hint to make the area more visible, but this is not a substitute for proper tagging features inside.

Here is the example of reserves with and without real tagging of features (when you zoom in, the green shade goes away, so it's easy to see if the tagging is complete):

[kocio-pl]

Low zoom areas in French fork are pre-rendered into one big (80 MB) TIFF file, which is then used as a data layer:

https://github.com/cquest/osmfr-cartocss/blob/master/get-layers.sh

https://github.com/cquest/osmfr-cartocss/blob/d6dc2a652fa6331d8a77ca11b8e563bd57539c13/osmfr.yml#L84

@cquest Could you tell how z7.tif is produced and how often do you update it?

[kocio-pl]

It might make sense to partially revert #2874 only for low zoom data layer. When I changed

AND way_area > 1*!pixel_width!::real*!pixel_height!::real

to be 0.01 instead of 1 again, I got following results:

z7 before (1)

z7 after (0.01)

z7 difference (compare -compose src pl-z7-1pixel.png pl-z7-001pixel.png difference.png)

[Adamant36]

@jeisenbe thanks for looking out for California ;)

@kocio-pl, I've been meaning to do an issue about Orchards being rendered at higher zoom levels. There's pretty large areas of them in California. Although a lot of lot them are mapped as small individual plots, but if they could be added anyway, that would be great. Otherwise, there's going to be pretty large gabs with the map in California.

[jeisenbe]

I’d recommend rendering orchards and vineyards the same as farmland at low zoom level, because they are all types of agricultural land.

…

On Thu, Oct 18, 2018 at 1:55 PM Adamant36 ***@***.***> wrote: @jeisenbe <https://github.com/jeisenbe> thanks for looking out for California ;) @kocio-pl <https://github.com/kocio-pl>, I've been meaning to do an issue about Orchards being rendered at higher zoom levels. There's pretty large areas of them in California. Although a lot of lot them are mapped as small individual plots, but if they could be added anyway, that would be great. Otherwise, there's going to be pretty large gabs with the map in California. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3458 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AoxshKDh_g-Weai9039-04rFoja2b6l8ks5umAnKgaJpZM4Xo4vW> .

[Adamant36]

It would definitely look better that way. I don't think the pattern on either one of them would work at higher zoom levels and the darker green of orchards might be confused for woods or something.

[Tomasz-W]

@kocio-pl Thanks for working on it! But I think it's way too light on z5, so I propose to test 2 other variants:

• current PR's z5 opacity level moved to z0 and fading to full opacity at z13
• opacity level between current PR's z5 and French style used for z5 and fading to full opacity at z 13

[kocio-pl]

I want this code to be very limited in scope. Low zoom is very specific and we have no big expertise on that, because most of our work focuses on high zoom levels (and occasionally medium ones).

Some interesting features of low zoom levels to consider:

• low zoom means big areas, which involves huge data sets, which makes it very hard to test anything above country level performance-wise (long time to import data, quite long waiting for rendering)
• there exist mainly natural objects, most human artifacts are too small (country borders are virtual, roads are exaggerated to see them at all)
• many objects are very unique there, so it's hard to find examples - instead of thousands of shops we have maybe dozens of such big wetlands and no easy way to pick them from database
• these are world-wide levels, so once tested, we have the full control and can even make some special tuning

What I want to do here is to find which objects make so big areas/clusters that would be visible on z5-z7 and make sure that performance is still acceptable. Which is quite contradicting - more objects and more (sub-pixel) accuracy create more pleasing image with less gaps, but at the same time we may end up reading tons of data which appear only in 2-3 places as a small visual patch, but eating all the server resources for hours.

Changing colors and tuning opacity is a valid issue, but this can be done once we have some solid knowledge what's going on. It's better to have faded colors and make them stronger later if needed than to create visual mess, so thanks to #2654 we're on the safe side now.

[kocio-pl]

Fresno in California looks better with vineyards and orchards added (z9):

Before

After

[kocio-pl]

California with vineyards, orchards and scrub using 0.01 px filter - z7 and z10 rendering (rescaled to match z7 rendering size with simply convert -resize 12.5% usa-ca-z10-001px.png usa-ca-z10-rescaled-001px.png from ImageMagick), so there are more small details not filtered out (also some big military areas and natural reserve borders):

z7

z10 rescaled

[kocio-pl]

Removing pixel limits for California on z7 did not make almost any visual difference, so it's accurate enough with >0.01 pixel filter:

[kocio-pl]

I believe the problem is that low zoom background image used in French fork is outdated, so our take is OK - this is current state of data.

When you zoom in from z7 into z8 you will see the differences near Fresno (forest appears and part of farmland disappears):

z7 (low zoom preprocessed data from image file)

z8 (current data)

[kocio-pl]

@rrzefox Could you apply this patch on your server? I'm especially interested in performance comparison for reredering z5-z9 in 3 cases:

• without this patch (current code)
• with this patch, but with 1 px limit
• with this patch (0.01 px limit)

Since this is a low zoom area this should be pretty deterministic and we will have data for all the tiles, not just some of them (this is the upside of testing low zoom levels).

[kocio-pl]

Just a remark - Antarctica's ice would be also visible from z5 instead of z8+:

[kocio-pl]

I try to measure performance using current Europe data, because @Komzpa server he offered for testing has not enough disc space for the whole planet and Europe still makes about half of all OSM data (it takes 447 GB in PostgreSQL), so it should be enough to get proper idea about the speed.

I run render_speedtest and it tries to render 2 tiles of every zoom level, however I don't know which are selected, so I'm not sure if the results will be reasonable (for example it won't try to render tiles outside Europe, where I have no data). I get it from here (OSMF uses these packages for its rendering servers):

https://launchpad.net/~osmadmins/+archive/ubuntu/ppa?field.series_filter=bionic

and it's included in package renderd, which has been developed as part of a mod_tile project:

https://github.com/openstreetmap/mod_tile
https://wiki.openstreetmap.org/wiki/Mod_tile

I guess this is the source of this application:

https://github.com/openstreetmap/mod_tile/blob/master/src/speedtest.cpp

Could anybody tell me if this test will give proper results?

[kocio-pl]

Initial batch - I had to halt the test after 2h, because it was eating almost full swap (RAM is 32 GB and swap is another 32 GB there):

Zoom(0) Now rendering 1 tiles
Rendered 1 tiles in 136.75 seconds (0.01 tiles/s)

Zoom(1) Now rendering 2 tiles
Rendered 2 tiles in 362.64 seconds (0.01 tiles/s)

Zoom(2) Now rendering 2 tiles
Rendered 2 tiles in 373.20 seconds (0.01 tiles/s)

Zoom(3) Now rendering 2 tiles
Rendered 2 tiles in 420.94 seconds (0.00 tiles/s)

Zoom(4) Now rendering 2 tiles
Rendered 2 tiles in 496.26 seconds (0.00 tiles/s)

Zoom(5) Now rendering 2 tiles
Rendered 2 tiles in 750.40 seconds (0.00 tiles/s)

Zoom(6) Now rendering 2 tiles
Rendered 2 tiles in 1173.62 seconds (0.00 tiles/s)

Zoom(7) Now rendering 2 tiles
Rendered 2 tiles in 1323.32 seconds (0.00 tiles/s)

[kocio-pl]

Second batch (cold reboot of renderd and postgresql):

Zoom(0) Now rendering 1 tiles
Rendered 1 tiles in 227.02 seconds (0.00 tiles/s)

Zoom(1) Now rendering 2 tiles
Rendered 2 tiles in 467.02 seconds (0.00 tiles/s)

Zoom(2) Now rendering 2 tiles
Rendered 2 tiles in 485.98 seconds (0.00 tiles/s)

Zoom(3) Now rendering 2 tiles
Rendered 2 tiles in 541.09 seconds (0.00 tiles/s)

Zoom(4) Now rendering 2 tiles
Rendered 2 tiles in 607.65 seconds (0.00 tiles/s)

Zoom(5) Now rendering 2 tiles
Rendered 2 tiles in 936.97 seconds (0.00 tiles/s)

Zoom(6) Now rendering 2 tiles
Rendered 2 tiles in 1399.52 seconds (0.00 tiles/s)

Zoom(7) Now rendering 2 tiles
Rendered 2 tiles in 1389.03 seconds (0.00 tiles/s)

[matthijsmelissen]

How does this compare with before?

[kocio-pl]

I just started with "after" (since this might be the worst case) - now starting third batch. Then I will test "before", also 3 times, so still many hours of testing ahead.

[kocio-pl]

3rd batch with 0.01 px limit (after cold reboot):

Zoom(0) Now rendering 1 tiles
Rendered 1 tiles in 223.89 seconds (0.00 tiles/s)

Zoom(1) Now rendering 2 tiles
Rendered 2 tiles in 453.27 seconds (0.00 tiles/s)

Zoom(2) Now rendering 2 tiles
Rendered 2 tiles in 471.73 seconds (0.00 tiles/s)

Zoom(3) Now rendering 2 tiles
Rendered 2 tiles in 527.09 seconds (0.00 tiles/s)

Zoom(4) Now rendering 2 tiles
Rendered 2 tiles in 598.08 seconds (0.00 tiles/s)

Zoom(5) Now rendering 2 tiles
Rendered 2 tiles in 881.99 seconds (0.00 tiles/s)

Zoom(6) Now rendering 2 tiles
Rendered 2 tiles in 1346.51 seconds (0.00 tiles/s)

Zoom(7) Now rendering 2 tiles
Rendered 2 tiles in 1371.35 seconds (0.00 tiles/s)

[kocio-pl]

First batch with 1 px - faster, but looks like this is still in the same range, which is good:

Zoom(0) Now rendering 1 tiles
Rendered 1 tiles in 190.01 seconds (0.01 tiles/s)

Zoom(1) Now rendering 2 tiles
Rendered 2 tiles in 388.43 seconds (0.01 tiles/s)

Zoom(2) Now rendering 2 tiles
Rendered 2 tiles in 402.12 seconds (0.00 tiles/s)

Zoom(3) Now rendering 2 tiles
Rendered 2 tiles in 463.05 seconds (0.00 tiles/s)    
           
Zoom(4) Now rendering 2 tiles
Rendered 2 tiles in 530.71 seconds (0.00 tiles/s)

Zoom(5) Now rendering 2 tiles
Rendered 2 tiles in 762.00 seconds (0.00 tiles/s)

Zoom(6) Now rendering 2 tiles
Rendered 2 tiles in 1027.49 seconds (0.00 tiles/s)

Zoom(7) Now rendering 2 tiles
Rendered 2 tiles in 1062.85 seconds (0.00 tiles/s)

[kocio-pl]

Second batch with 1 px:

Zoom(0) Now rendering 1 tiles
Rendered 1 tiles in 199.03 seconds (0.01 tiles/s)

Zoom(1) Now rendering 2 tiles
Rendered 2 tiles in 399.15 seconds (0.01 tiles/s)

Zoom(2) Now rendering 2 tiles
Rendered 2 tiles in 420.25 seconds (0.00 tiles/s)

Zoom(3) Now rendering 2 tiles
Rendered 2 tiles in 489.98 seconds (0.00 tiles/s)

Zoom(4) Now rendering 2 tiles
Rendered 2 tiles in 561.41 seconds (0.00 tiles/s)

Zoom(5) Now rendering 2 tiles
Rendered 2 tiles in 814.08 seconds (0.00 tiles/s)

Zoom(6) Now rendering 2 tiles
Rendered 2 tiles in 1108.51 seconds (0.00 tiles/s)

Zoom(7) Now rendering 2 tiles
Rendered 2 tiles in 1151.34 seconds (0.00 tiles/s)

[kocio-pl]

This code does some basic things, precisely shows some natural areas earlier. How should they look like however is a different question, which is related to the recent ticket #3513. Also any gamma tuning should be researched later.

In this code I have decreased military minzoom from z7 to z8 - it's not directly related to natural areas, but it's a performance fix, which can make the problem of rendering more elements on low zoom less resource hungry.

[matthijsmelissen]

Has this change been approved by another maintainer? I think this vhange has both advantages and disadvantages, and would need more careful consideration.

Unfortunately myself I’m in the middle of moving houses myself, and don’t have a lot of spare time at the moment.

[kocio-pl]

I was waiting for comments as usual and there was only one simple question from you, so I'm surprised that you haven't even mentioned about possible disadvantages before.

[jeisenbe]

I think there should have been time to comment on the last two commit. It looks like the change is removing military areas from z7 in the PR, so now they will continue to render only from z8, as before this PR?

It would be good to show military areas and national parks earlier if we are going to show other landuse/landcover and natural areas:

Eg Northern Territory, Australia:

z8 Darwin

z8 South of Darwin

z7 Without Military Areas (but with national parks)

z7 With Military and National Parks

Unfortunately I don't have any large USA States downloaded, but I know there are several very large military areas in the western USA and Alaska, which should render on z8

Perhaps this can be added with the next PR to address this issue?

This may not be the right thread, but I would also support maintainers not merging their own PRs, for fairness and better oversight.

[kocio-pl]

Sorry, I was probably not clear enough - nothing is changed with rendering military areas. The database is currently being asked about military data on z7, but nothing is rendered there, so it was just a fix for hidden database abuse.

Perhaps this can be added with the next PR to address this issue?

It's just a step toward low zoom improvements (https://github.com/gravitystorm/openstreetmap-carto/projects/4), however we have some guidelines already:

https://github.com/gravitystorm/openstreetmap-carto/blob/master/USECASES.md

This may not be the right thread, but I would also support maintainers not merging their own PRs, for fairness and better oversight.

Yes, the proper thread is here: #2436.

In short - it would be better, but we need more active developers with real merging rights willing to peer review the code. Currently these are mostly 2 different groups, but I hope new developers can get "collaborator" status eventually or some old developers will start making reviews again.

[jragusa]

This PR looks good to me

[Tomasz-W]

Whoa...

😍

@kocio-pl Thanks! Now I'm looking forward to natural areas on z0-z4 & changing borders colour :)

[kocio-pl]

This change shows that borders are getting important now. This green and violet mix is not as bad as in French fork, but it's disturbing for me.

I like this place the most - I didn't know it's there at all, but it gives a context explaining how cities are placed around: