feat(lambda-python): support for providing a custom bundling docker image #18082

setu4993 · 2021-12-18T10:31:44Z

This refactors the bundling process to match the NodeJs and Go Lambda functions and allows providing a custom bundling docker image.

Changes:

refactor bundling to use cdk.BundlingOptions
Use updated Bundling class
Update tests to use updated Bundling class

Fixes #10298, #12949, #15391, #16234, #15306

BREAKING CHANGE: assetHashType and assetHash properties moved to new bundling property.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

gitpod-io · 2021-12-18T10:31:46Z

ranjith-jagadeesh

By mistake I have approved

packages/@aws-cdk/aws-lambda-python/lib/bundling.ts

setu4993 · 2021-12-25T09:27:49Z

packages/@aws-cdk/aws-lambda-python/test/integ.function.requirements.removed.ts

-// Then, write a requirements.txt without the extraneous dependency and synth again
-const afterDeps = 'certifi==2020.6.20\nchardet==3.0.4\nidna==2.10\nurllib3==1.26.7\nrequests==2.26.0\nPillow==8.4.0\n';
-fs.writeFileSync(requirementsTxtPath, afterDeps);


This test kept failing for me, regardless of how I modified the afterDeps value. In the traceback, I could only see one of the commands being run, which makes me think it has something to do with how synth + caching occurs with the bundling process.

In the previous version, dependency installation was occurring during the Docker build process (compared to Docker image run now), so maybe that has something to do with it? Not sure.

Noticed this in the logs:

### `cdk synthesize` aws-cdk: Synthesizes the CDK app and produces a cloud assembly to a designated output (defaults to `cdk.out`) aws-cdk: Typically you don't interact directly with cloud assemblies. They are files that include everything aws-cdk: needed to deploy your app to a cloud environment. For example, it includes an AWS CloudFormation aws-cdk: template for each stack in your app, and a copy of any file assets or Docker images that you reference aws-cdk: in your app.

which to me reads like: app.synth() in the above referenced stack wouldn't actually create the asset; that would be during cdk deploy.

setu4993 · 2021-12-25T10:14:24Z

@corymhall : Given all tests, including integration tests are succeeding, I feel good about this PR. I'm happy to start documenting this into the readme and rebase to master if this feels like a good direction.

corymhall · 2021-12-27T13:40:51Z

@corymhall : Given all tests, including integration tests are succeeding, I feel good about this PR. I'm happy to start documenting this into the readme and rebase to master if this feels like a good direction.

Sounds good.

setu4993

@corymhall : This is complete and ready for review.

setu4993 · 2021-12-28T06:33:56Z

packages/@aws-cdk/aws-lambda-python/README.md

+```ts
+new lambda.PythonFunction(this, 'MyFunction', {
+  entry: '/path/to/my/function',
+  runtime: Runtime.PYTHON_3_6,
+  layers: [
+    new lambda.PythonLayerVersion(this, 'MyLayer', {
+      entry: '/path/to/my/layer', // point this to your library's directory
+    }),
+  ],
+});
+```


Moved this up.

setu4993 · 2021-12-28T06:34:27Z

packages/@aws-cdk/aws-lambda-python/README.md

+```ts
+new lambda.PythonLayerVersion(this, 'MyLayer', {
+  entry: '/path/to/my/layer', // point this to your library's directory
+})


Added this example.

setu4993 · 2021-12-28T06:35:01Z

packages/@aws-cdk/aws-lambda-python/README.md

+Packaging is executed using the `Packaging` class, which:
+
+1. Infers the packaging type based on the files present.
+2. If it sees a `Pipfile` or a `poetry.lock` file, it exports it to a compatible `requirements.txt` file with credentials (if they're available in the source files or in the bundling container).
+3. Installs dependencies using `pip`.
+4. Copies the dependencies into an asset that is bundled for the Lambda package.


Added some details on how bundling occurs.

setu4993 · 2021-12-28T06:35:25Z

packages/@aws-cdk/aws-lambda-python/README.md

+## Custom Bundling
+
+Custom bundling can be performed by using custom Docker images for bundling dependencies. If using a custom Docker image for bundling, the dependencies are installed with `pip`, `pipenv` or `poetry` by using the `Packaging` class.
+
+A different bundling Docker image that is in the same directory as the function can be specified as:
+
+ ```ts
+const entry = '/path/to/function';
+const image = DockerImage.fromBuild(entry);
+
+new PythonFunction(stack, 'function', {
+  entry,
+  bundling: { image },
 });


Added example that shows how to use a custom bundling image.

setu4993 · 2021-12-28T06:35:48Z

packages/@aws-cdk/aws-lambda-python/lib/Dockerfile

-# Ensure rsync is installed
-RUN yum -q list installed rsync &>/dev/null || yum install -y rsync
+# Upgrade pip (required by cryptography v3.4 and above, which is a dependency of poetry)
+RUN pip install --upgrade pip
+RUN pip install pipenv poetry


We don't need rsync and can just use cp.

setu4993 · 2021-12-28T06:42:02Z

packages/@aws-cdk/aws-lambda-python/lib/bundling.ts

+      },
+      platform: architecture.dockerPlatform,
+    });
+    this.image = image ?? defaultImage;


This is where a custom Docker image would get pulled in when bundling.

setu4993 · 2021-12-28T06:42:30Z

packages/@aws-cdk/aws-lambda-python/lib/function.ts

   */
-  readonly assetHashType?: AssetHashType;
+  readonly runtime?: Runtime;


Making this optional.

We decided to make this a required property in #18143

Oh I missed that, thanks for the tip! Wouldn't it be nicer to default to Python 3.8 instead to better resolve #10248? That has a much longer EOL, has been around for a few years and is well supported.

setu4993 · 2021-12-28T06:43:33Z

packages/@aws-cdk/aws-lambda-python/lib/function.ts

   */
-  readonly assetHashType?: AssetHashType;


Moved to BundlingOptions.

setu4993 · 2021-12-28T06:43:40Z

packages/@aws-cdk/aws-lambda-python/lib/function.ts

   */
-  readonly assetHash?: string;


Moved to BundlingOptions.

setu4993 · 2021-12-28T06:44:21Z

packages/@aws-cdk/aws-lambda-python/lib/function.ts

-    if (props.runtime && props.runtime.family !== lambda.RuntimeFamily.PYTHON) {
-      throw new Error('Only `PYTHON` runtimes are supported.');
-    }
+    const { index = 'index.py', handler = 'handler', runtime = Runtime.PYTHON_3_7 } = props;


Defining defaults while unpacking props.

…nd layers

corymhall

@setu4993 This looks awesome! Thanks for all the work on this, we're closing a lot of issues :)

mergify · 2021-12-30T13:34:29Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2021-12-30T14:29:46Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
Commit ID: b94dbac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2021-12-30T14:30:09Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

setu4993 · 2021-12-31T01:34:31Z

@setu4993 This looks awesome! Thanks for all the work on this, we're closing a lot of issues :)

Thanks @corymhall! Glad we got this in place :).

chrispykim · 2022-01-09T23:09:03Z

Hi @setu4993, sorry if this isn't an appropriate place to raise this, but I am having issues with my PythonFunction since this change.

my_lambda = lambda_python.PythonFunction(self, "MyLambdaName",
            entry=path.join(this_dir, "lambda_with_dependencies"),
            index="hello_lambda.py",
            runtime=lambda_.Runtime.PYTHON_3_8,
            log_retention=logs.RetentionDays.THREE_MONTHS,
            reserved_concurrent_executions=1,
            timeout=core.Duration.seconds(60),
            environment={
                "SOMEVAR": "some value"
            })

This is my CDK code, which worked perfectly fine with CDK 1.137.0. With 1.138.0 (and 1.138.1), without making any changes to the above CDK code or the directory structure or the lambda implementation, my lambda suddenly stopped working with this error:

{
  "errorMessage": "Unable to import module 'hello_lambda': No module named 'hello_lambda'",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

Perhaps I am not fully understanding the implications of the breaking changes, but my understanding is that my code should still set up the Docker container just as it did previously. Can you point me towards a solution that works with CDK 1.138.0 and above?

setu4993 · 2022-01-10T00:20:28Z

@chrispykim : Thanks for flagging! I missed a /* in there, which was also flagged in #18301 and attempted to fix in #18306, but can confirm it wasn't resolved for me in "1.138.1". I have a fix for it open in #18335, which I have tested locally.

@chrispykim

Asset files are incorrectly being bundled under the `asset-input` directory instead of the root of the bundle. To also copy over hidden files (#18306 (comment)), I switched from using `-R` to `-a` based on what I found on [SO](https://stackoverflow.com/a/13020113) and the [man page](https://linux.die.net/man/1/cp). (`-a` is equivalent to `-dR`.) Fixes #18301 and @chrispykim's comment: #18082 (comment). ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

…mage (aws#18082) This refactors the bundling process to match the NodeJs and Go Lambda functions and allows providing a custom bundling docker image. Changes: - refactor bundling to use `cdk.BundlingOptions` - Use updated `Bundling` class - Update tests to use updated `Bundling` class Fixes aws#10298, aws#12949, aws#15391, aws#16234, aws#15306 BREAKING CHANGE: `assetHashType` and `assetHash` properties moved to new `bundling` property. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

@chrispykim

Asset files are incorrectly being bundled under the `asset-input` directory instead of the root of the bundle. To also copy over hidden files (aws#18306 (comment)), I switched from using `-R` to `-a` based on what I found on [SO](https://stackoverflow.com/a/13020113) and the [man page](https://linux.die.net/man/1/cp). (`-a` is equivalent to `-dR`.) Fixes aws#18301 and @chrispykim's comment: aws#18082 (comment). ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

…19143) #18082 added the ability to pass bundling parameters to `PythonFunction`'s. Currently when the `image` parameters is passed both the passed image and the default image are built even though the default image is not used ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions bot assigned corymhall Dec 18, 2021

github-actions bot added the @aws-cdk/aws-lambda-python label Dec 18, 2021

setu4993 force-pushed the feat/lambda-python/improve-bundling branch from 91baabe to ca10615 Compare December 18, 2021 10:34

ranjith-jagadeesh approved these changes Dec 18, 2021

View reviewed changes

ranjith-jagadeesh reviewed Dec 18, 2021

View reviewed changes

corymhall reviewed Dec 20, 2021

View reviewed changes

packages/@aws-cdk/aws-lambda-python/lib/bundling.ts Outdated Show resolved Hide resolved

packages/@aws-cdk/aws-lambda-python/lib/bundling.ts Outdated Show resolved Hide resolved

setu4993 force-pushed the feat/lambda-python/improve-bundling branch from d55c1f6 to 5bcab84 Compare December 25, 2021 09:23

setu4993 commented Dec 25, 2021

View reviewed changes

setu4993 force-pushed the feat/lambda-python/improve-bundling branch from 5bcab84 to 0a43b76 Compare December 28, 2021 01:58

setu4993 changed the title ~~feat(lambda-python): Improve bundling to align with other Lambda functions, support custom bundling Docker image~~ feat(lambda-python): Refactor bundling to align with other Lambda functions, support custom bundling Docker image Dec 28, 2021

setu4993 commented Dec 28, 2021

View reviewed changes

setu4993 added 16 commits December 28, 2021 15:52

feat: refactor bundling to use cdk.BundlingOptions

6bc24d1

fix: Use updated Bundling class

1d018bb

test: Update tests to use updated Bundling class

4123034

feat: Add new Packaging class

69ebdaa

refactor: Bundling to use Packaging, align with other Lambda functions

97f1509

test: Add tests for Packaging

d9b306e

test: Add, update bundling tests

a4a2feb

test: Add, update function, layer tests

b0d9432

fix: Remove integ test that isn't applicable

4e7868f

feat: Split, export BundlingOptions

00f8bdb

test: Fix function tests

bddeaa5

test: Add bundling, integ test for custom Docker image

21ddb3b

test: Ignore reqs.txt generated during test runs

f8e2a14

fix: Add default value for Runtime

a6ddd7e

feat: Add bundling as an option for PythonLayerVersion

8fadde5

test: Add tests for validating custom images are used for functions a…

4357764

…nd layers

setu4993 requested a review from corymhall December 30, 2021 00:28

setu4993 added 2 commits December 29, 2021 17:13

docs: PythonFunction -> lambda.PythonFunction

3f73b0a

fix: Set region explicitly

5e48214

corymhall changed the title ~~feat(lambda-python): Refactor bundling to align with other Lambda functions, support custom bundling Docker image~~ feat(lambda-python): support for providing a custom bundling docker image Dec 30, 2021

corymhall approved these changes Dec 30, 2021

View reviewed changes

Merge branch 'master' into feat/lambda-python/improve-bundling

b94dbac

mergify bot merged commit c3c4a97 into aws:master Dec 30, 2021

setu4993 deleted the feat/lambda-python/improve-bundling branch December 31, 2021 01:34

setu4993 mentioned this pull request Jan 10, 2022

fix(lambda-python): bundle asset files correctly #18335

Merged

njlynch mentioned this pull request Jan 10, 2022

‼️ (aws-lambda-python): SAM - CDK integration is broken in CDK V2.4.0 #18301

Closed

relm923 mentioned this pull request Feb 24, 2022

fix(aws-lambda-python): skip default docker build when image passed #19143

Merged

corymhall mentioned this pull request Mar 8, 2022

(aws-lambda-python): skip build lambda for all cdk commands #12684

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(lambda-python): support for providing a custom bundling docker image #18082

feat(lambda-python): support for providing a custom bundling docker image #18082

setu4993 commented Dec 18, 2021 •

edited by corymhall

Loading

gitpod-io bot commented Dec 18, 2021

ranjith-jagadeesh left a comment

setu4993 Dec 25, 2021

setu4993 Dec 25, 2021

setu4993 commented Dec 25, 2021

corymhall commented Dec 27, 2021

setu4993 left a comment

setu4993 Dec 28, 2021

setu4993 Dec 28, 2021

setu4993 Dec 28, 2021

setu4993 Dec 28, 2021

setu4993 Dec 28, 2021

setu4993 Dec 28, 2021

setu4993 Dec 28, 2021

corymhall Dec 29, 2021

setu4993 Dec 29, 2021

setu4993 Dec 28, 2021

setu4993 Dec 28, 2021

setu4993 Dec 28, 2021

corymhall left a comment

mergify bot commented Dec 30, 2021

aws-cdk-automation commented Dec 30, 2021

mergify bot commented Dec 30, 2021

setu4993 commented Dec 31, 2021

chrispykim commented Jan 9, 2022 •

edited

Loading

setu4993 commented Jan 10, 2022

feat(lambda-python): support for providing a custom bundling docker image #18082

feat(lambda-python): support for providing a custom bundling docker image #18082

Conversation

setu4993 commented Dec 18, 2021 • edited by corymhall Loading

gitpod-io bot commented Dec 18, 2021

ranjith-jagadeesh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

setu4993 commented Dec 25, 2021

corymhall commented Dec 27, 2021

setu4993 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corymhall left a comment

Choose a reason for hiding this comment

mergify bot commented Dec 30, 2021

aws-cdk-automation commented Dec 30, 2021

AWS CodeBuild CI Report

mergify bot commented Dec 30, 2021

setu4993 commented Dec 31, 2021

chrispykim commented Jan 9, 2022 • edited Loading

setu4993 commented Jan 10, 2022

setu4993 commented Dec 18, 2021 •

edited by corymhall

Loading

chrispykim commented Jan 9, 2022 •

edited

Loading