AWS Api Gateway - Tile Response Encoding

[DanSchoppe]

Hi @vincentsarago,

Thanks as always for all you do for this community 🙏

I've been using a Titiler Lambda integrated with an API Gateway endpoint since August. After attempting to upgrade from v0.3.3 to v0.3.12 (I want to use the new /statistics endpoint!! 😄), my tile requests from both the /stac/tiles/... and /cog/tiles/... are giving me trouble. The data is coming back Base64 encoded, rather than binary:

You can see that when I Base64 decode the data, I have a valid JPEG:

Obviously API Gateway settings are a bit finicky regarding this type of thing, but I don't believe that's what's at fault in this case. I version control all of my infrastructure, and a change to the Titiler version alone seems to be responsible for this change in behavior.

After looking through the diffs over the last couple months, I noticed the upgrade from v0.3.9 -> v0.3.10 included a change to response middleware. Do you think that could be the culprit? Any other change you can think of that might have caused such a thing?

Thanks for your thoughts,
Dan

[vincentsarago]

That's really interesting.

So the change between 0.3.9 and 0.3.10 you are referring is the switch to starlette_cramjam, which should return the exact same data as the previous compression middleware.

can you check the headers returned by the API?

[DanSchoppe]

Yeah, super strange. Thanks for chiming in.

Headers:

date: Fri, 29 Oct 2021 14:45:47 GMT
content-type: image/jpeg
content-length: 956
x-amzn-requestid: ...
x-amzn-remapped-content-length: 717
x-amz-apigw-id: ...
cache-control: public, max-age=3600
x-amzn-trace-id: Root=...;Sampled=0

I'm not familiar with x-amzn-remapped-content-length, but the fact that it's reporting a different value than content-length seems suspiciously like API Gateway is mutating the response. In fact, if I Base64 decode the 956-byte response payload, the result is a 717 byte (matches x-amzn-remapped-content-length!) well-formed JPEG. So I'm guessing that this is likely user error and I need to dig deeper into why API Gateway is encoding my payload.

This Lambda is the result of building Titiler myself and hooking it up to API GW, rather than using the provided AWS CDK:

git clone https://github.com/developmentseed/titiler.git
cd titiler
git checkout 0.3.12
cd deployment/aws
docker build -f ./lambda/Dockerfile -t titiler:0.3.12 .
mkdir asset-output
docker run --rm -it -v $(pwd)/asset-output/:/asset-output titiler:0.3.12 bash -c 'cp -R /asset/. /asset-output/.'
cd asset-output
zip -r ../package.zip .
aws s3 cp ../package.zip s3://[redacted]/titiler/v0.3.12.zip --profile admin

That also means that the API Gateway definition is my own as well. All Lambda and API GW settings are inspired by values in your CDK definition, but it sure seems likely that the root cause is in my infrastructure definition.

I'll keep digging and report back. @vincentsarago Do you happen to know of any API Gateway-hosted instances of ^0.3.10 that have properly functioning [cog,stac]/tiles endpoints? That would probably definitively indicate that this is my fault.

Thanks again,
Dan

[vincentsarago]

I'll keep digging and report back. @vincentsarago Do you happen to know of any API Gateway-hosted instances of ^0.3.10 that have properly functioning [cog,stac]/tiles endpoints? That would probably definitively indicate that this is my fault.

I had one until this morning 😬, I could try to deploy one over the week end.

[DanSchoppe]

This looks to be a problem with missing Binary Media Types specified in my API Gateway REST API settings. A new */* wildcard entry fixes the issue. I'm a bit baffled why I didn't previously need any binary media types specified with Titiler v0.3.3, but 🤷 I'm going to simply move ahead with a new */* entry without digging any further into root cause.

@vincentsarago thanks for your attention on this and I'm sorry for the distraction. I've still got the previous configuration around in a non-prod environment if this is something interesting to you and you'd care to see any request/responses.

[vincentsarago]

@DanSchoppe can you confirm you are not using the default CDK code provided in the repo?

[DanSchoppe]

That's correct, I am not using the CDK provided in this repo.

[vincentsarago]

👍