Allow downloading of mbtiles per task

[susmina94]

Problem:
The mbtiles are too large and takes up more memory space if I download the entire mbtile of the project area.

Solution:
I want to download only the mbtiles of assigned task
This will save my memory space and easy to load.

[robsavoye]

Considering multiple people are generating mbtiles, on the server side it should download all tiles in the project AOI, and then produce the smaller mbtiles for each task. I assume you are having a problem doing this on a local machine, and not the dev server ?

[spwoodcock]

I have a PR in place to download different formats (mbtiles, pmtiles, sqlite).
Currently adding another PR to store tiles in S3 instead of locally.

Proposed Workflow

Online:

• On project creation, all tiles are downloaded, packaged as PMTiles and stored in S3.
• The PMTiles can be used as the basemap for online project viewing.

Offline:

• User selects to download a subset of tiles to MBTile format, either via selecting a few task boundaries, or the entire project if they like.
• Tiles in the bbox are extracted out of the PMTiles archive into MBTiles format for offline use by user.
  (either using new pmtiles extract CLI Add pmtiles extract command protomaps/go-pmtiles#31 or a re-implementation in Python).

[robsavoye]

There is an inbetween online & offline. I have plenty of memory and storage, so I'd be working online, but not using S3 at all.

[spwoodcock]

Is this complete?

I didn't think so.

We need code to split an mbtiles or pmtiles for an entire project into individual task areas.

@manjitapandey

[spwoodcock]

@robsavoye suggested we can just run basemapper for the whole aoi, then again for each individual task area.

It should used the cached tiles on disk for the task areas, so will be very quick to complete.

This is a great idea & saves writing code for pmtile --> mbtile conversion.

[spwoodcock]

Also, to do this, we need to generate all of the task area mbtiles in one go automatically (when a project is created?)

This is probably a good idea anyway, as long as it doesn't take up a large amount of S3 storage space.

[nrjadkry]

@spwoodcock
I will be working in it. Probably tomorrow or next week.
One question is, what source should we use for default when we generate mbtiles when creating a project.

[robsavoye]

By default, I use ESRI.

[spwoodcock]

Excellent, thanks @nrjadkry

My thoughts are to:

• Generate the entire AOI as PMTiles when a project is created.
• Using the same tile cache, generate the MBTile files for each individual task.

Both can be done as a background task.

The advantage of the PMTiles is that it can be streamed when the user is online.

The MBTiles can be stored in the S3 and downloaded as needed before going offline.

Perhaps:
/orgid/projectid/basemap.pmtiles
/orgid/projectid/basemaps/taskid.mbtiles

[robsavoye]

We don't want to cache the map tiles under org or project, as we'll get duplication. For the generated mbtiles basemap, that does make send to store them under project id. Not sure if org is needed.

[spwoodcock]

Yeah we don't want to store the cached tiles - they get discarded afterwards and we only keep the generated tile archives (we should probably add the step to delete the tile output folder from the tmp dir after we have all the tile archives generated 👍)

As for the org id, it's probably not necessary, true - we already have the folder structure in place, but could consider changing it in the future (I thought the org id made the bucket root dir a but neater, rather than 100's-1000's of project dirs in one).

[spwoodcock]

Storage on AWS is approx $0.08/GB, so 300GB of data would cost $288 a year per instance.

Make the data global and for multiple providers, then we are talking $1000's spent on storing map tiles.

[susmina94]

@NSUWAL123 I think Sujan has already prepared an api for downloading tiles per task and as per the last discussion, we will be generating tiles using ESRI imagery by default.

But we do not have any UI on the frontend side for the users to download the tiles per task. I will suggest a design for the pop-up that includes the download icon and you can proceed ahead.

[spwoodcock]

I closed #1080 for now.

Question

• Do we have any field-verified evidence that this is useful?
• Have any users complained about the size of the mbtiles for a project area being too big?
• I think that having an mbtiles file per task would be unwieldy quite quickly in the ODKCollect UI (it's just not made for that).

If the answer is, 'no we don't have requests for this', then I think we should close until a future time when (if) we do have requests.

Possible Solution

• If we decide this is useful, instead of pre-generating mbtiles per task, instead I think this should be done on-the-fly as it seems like more of a niche requirement.
• The mbtiles should be hosted on S3 (we should make an issue for this part!), then it can be passed through another API.
• We could have an API to grab the S3 mbtiles, extract out exactly the images in the AOI we need, then update the underlying SQLite file in the mbtiles to only include the task area images.

[spwoodcock]

Closing this until we find a good reason to implement it.

It can always be re-opened if someone discovers that 👍