Skip to content

Commit

Permalink
Implement http pmtiles (#991)
Browse files Browse the repository at this point in the history
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.

PMTiles config now also allows `http` and `https` protocol.

```
# Publish PMTiles files
pmtiles:
  paths:
    # specific pmtiles file will be published as mypmtiles source
    # (use last portion of the URL without extension)
    - http://example.org/path/to/mypmtiles.pmtiles
  sources:
    # named source matching source name to a single file
    pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
 ```

fixes #884

---------

Co-authored-by: Kyle Slugg-Urbino <[email protected]>
  • Loading branch information
nyurik and kyleslugg authored Dec 22, 2023
1 parent 30491ae commit 1a8e7c8
Show file tree
Hide file tree
Showing 27 changed files with 902 additions and 231 deletions.
84 changes: 82 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,8 @@ jobs:
tool: cross
- name: Checkout sources
uses: actions/checkout@v4
with:
set-safe-directory: false
- uses: Swatinem/rust-cache@v2
if: github.event_name != 'release' && github.event_name != 'workflow_dispatch'
- name: Init database
Expand Down Expand Up @@ -162,6 +164,13 @@ jobs:
mkdir -p target_releases/linux/amd64
mv target_releases/x86_64-unknown-linux-musl/* target_releases/linux/amd64/
- name: Start HTTP Server
run: |
docker run --rm -d \
--name fileserver \
-p 5412:80 \
-v ${{ github.workspace }}/tests/fixtures/pmtiles2:/usr/share/nginx/html \
nginx:alpine
- name: Build linux/arm64 Docker image
uses: docker/build-push-action@v5
# https://github.com/docker/build-push-action
Expand Down Expand Up @@ -229,6 +238,9 @@ jobs:
tags: ${{ steps.docker_meta.outputs.tags }}
labels: ${{ steps.docker_meta.outputs.labels }}
platforms: linux/amd64,linux/arm64
- name: Stop HTTP Server
if: always()
run: docker stop fileserver

build:
name: Build ${{ matrix.target }}
Expand Down Expand Up @@ -302,6 +314,61 @@ jobs:
uses: actions/checkout@v4
- uses: Swatinem/rust-cache@v2
if: github.event_name != 'release' && github.event_name != 'workflow_dispatch'
- name: Install Docker (MacOS)
if: runner.os == 'macos'
run: brew install docker && colima start
- name: Start HTTP Server (with Docker)
if: runner.os != 'windows'
run: |
docker run --rm -d \
--name fileserver \
-p 5412:80 \
-v ${{ github.workspace }}/tests/fixtures/pmtiles2:/usr/share/nginx/html \
nginx:alpine
- name: Start HTTP Server (Windows, no Docker)
if: runner.os == 'windows'
shell: pwsh
run: |
$nginxConf = @"
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 5412;
server_name localhost;
location / {
root $($PWD.Path)/tests/fixtures/pmtiles2;
index index.html index.htm;
}
}
}
"@
# Replace the default Nginx configuration file
echo "$nginxConf"
Set-Content -Path "C:\tools\nginx-1.25.3\conf\nginx.conf" -Value $nginxConf
Get-Content -Path "C:\tools\nginx-1.25.3\conf\nginx.conf"
# Start Nginx
#Get-Service -ErrorAction SilentlyContinue
#Get-CimInstance -ClassName Win32_Service
Set-Service nginx -StartupType manual
Start-Service nginx
#Start-Process -FilePath "C:\tools\nginx-1.25.3\nginx.exe"
dir C:\tools\nginx-1.25.3\logs\
Start-Sleep -Seconds 5
netstat -a
# Print Nginx Error Logs (on Windows systems)
#nginx -t
Get-Content -Path "C:\tools\nginx-1.25.3\logs\error.log"
dir D:\a\martin\martin\
- name: Start postgres
uses: nyurik/[email protected]
id: pg
Expand Down Expand Up @@ -355,7 +422,10 @@ jobs:
tests/test.sh
env:
DATABASE_URL: ${{ steps.pg.outputs.connection-uri }}
- name: Save test output on failure
- name: Stop HTTP Server (with Docker)
if: runner.os != 'windows'
run: docker stop fileserver
- name: Save test output (on error)
if: failure()
uses: actions/upload-artifact@v3
with:
Expand Down Expand Up @@ -422,6 +492,13 @@ jobs:
uses: actions/checkout@v4
- uses: Swatinem/rust-cache@v2
if: github.event_name != 'release' && github.event_name != 'workflow_dispatch'
- name: Start HTTP Server
run: |
docker run --rm -d \
--name fileserver \
-p 5412:80 \
-v ${{ github.workspace }}/tests/fixtures/pmtiles2:/usr/share/nginx/html \
nginx:alpine
- name: Init database
run: tests/fixtures/initdb.sh
env:
Expand Down Expand Up @@ -483,7 +560,10 @@ jobs:
cargo clean
env:
DATABASE_URL: postgres://${{ env.PGUSER }}:${{ env.PGUSER }}@${{ env.PGHOST }}:${{ job.services.postgres.ports[5432] }}/${{ env.PGDATABASE }}?sslmode=${{ matrix.sslmode }}
- name: On error, save test output
- name: Stop HTTP Server
if: always()
run: docker stop fileserver
- name: Save test output (on error)
if: failure()
uses: actions/upload-artifact@v3
with:
Expand Down
Loading

0 comments on commit 1a8e7c8

Please sign in to comment.