Building AWS Lambda layers for Python

[miikka]

Pants has good-looking support for building Python bundles for AWS Lambda. However, I'm wondering if there's an easy way to also build lambda layers. They're used for sharing code between lambdas and they're basically zip files of code with specific structure. Building them is similar but not quite the same as building lambda bundles.

The use case I have in mind is that we have a bunch of lambdas that use the same set of dependencies, which we have bundled into a layer. This makes the bundle sizes smaller and deployments faster. Currently we're building the layer with AWS CDK's PythonLayerVersion construct, which we also use to deploy them, but it slows down our CDK runs.

[bobthemighty]

I'm not a pants maintainer, but I'm on a similar journey!

At present, I'm just building layers separately, by zipping up site-packages, but I'm interested in revamping Pants' support for lambda, and writing a plugin for the serverless framework that uses pants to package both layers and functions.

The current python_lambda_function target creates a function wrapped with Lambdex. This approach is unofficially deprecated. I've been building lambda functions with the pex_binary target

pex_binary (
    name="main",
    layout="zipapp",
    inherit_path='fallback',
    
    dependencies=[
        "src/handler.py",
    ]
)

To run the pexified code, you need to append the __pex__ module to your lambda handler, eg. if your handler is src.handler.hello_world then you use __pex__.src.handler.hello_world. You'll get a warning on startup unless you set the environment variable PEX_ROOT=/tmp.

I've hit a couple of issues with lambda pex files resolving dependencies from layers. I've managed to get my scikit-learn layer to run by setting the env vars PYTHONPATH=/var/task and PEX_EXTRA_SYS_PATH=/opt/python.

Let me know if you get any problems, cos I'm slowly building a mental picture of how a lambda target should work.

[bobthemighty]

Actually, I clean forgot that I already wrote a prototype plugin for building lambda layers with Pants: https://github.com/bobthemighty/pants-lambda-layer

I haven't actually used this in production, because for my use cases I need to build a bunch of libraries from source, but if you're wanting to ship a layer full of ordinary Python modules, it might fit your needs. Feel free to open an issue on that repo if it doesn't work.

[miikka]

Why is the Lambdex approach being deprecated? Are there some problems with it?

[bobthemighty]

I'm not an expert here, but I think it's another thing to consider during maintenance, and it's not necessary, because PEX has evolved to be usable in a Lambda context. I think Lambdex is a shim to plug some gaps that no longer exist.

[jsirois]

@bobthemighty has that completely right.

[huonw]

In 2.18, there's now native support for a python_aws_lambda_layer target: https://github.com/pantsbuild/pants/blob/release_2.18.0.dev1/docs/markdown/Python/python-integrations/awslambda-python.md#building-a-lambda-layer

NB. this is currently a very early dev release (2.18.0.dev1 at the time of writing), which means potential instability, but is also the easiest time to implement improvements. If you happen to have time to test and provide feedback, that'd be great!

There's some other improvements to Lambdas happening in Pants 2.17 and 2.18 (#19076), so I'd recommend explicitly doing it in three steps, and running pants package ... etc. on an appropriate subset of your repo after each one, to resolve any changes, rather than skipping straight to 2.18:

1. upgrade to Pants 2.16 (pants_version = "2.16.0rc5" is the latest, at the time of writing)
2. upgrade to Pants 2.17 (pants_version = "2.17.0rc0"), following https://github.com/pantsbuild/pants/blob/release_2.17.0rc0/docs/markdown/Python/python-integrations/awslambda-python.md#migrating-from-pants-216-and-earlier
3. upgrade to Pants 2.18 (pants_version = "2.18.0.dev1"), following https://github.com/pantsbuild/pants/blob/release_2.18.0.dev1/docs/markdown/Python/python-integrations/awslambda-python.md#migrating-from-pants-216-and-earlier

There'll be a bunch more dev/alpha/RC releases for 2.18, and 2.16 (and 2.17) will hopefully be out soon, so experimenting will be smoother soon.

[huonw]

I've provided more info in #19544 for testing the new features in Pants' AWS Lambda (and GCF) support.