(aws-lambda-python): Using poetry requirements results in requirements.txt written to 'entry' directory #19231

huonw · 2022-03-03T22:10:26Z

What is the problem?

We're starting to use PythonFunction from @aws-cdk/aws-lambda-python-alpha with poetry (a pyproject.toml and poetry.lock file in the directory specified by entry), and it works pretty well, thanks.

However, the exported requirements.txt is written to the asset input volume directory, meaning it ends up written to the entry directory next to pyproject.toml, which is a bit unexpected. This requires configuring .gitignore correctly, and also may impact the fingerprinting procedure.

aws-cdk/packages/@aws-cdk/aws-lambda-python/lib/packaging.ts

Line 49 in 6dfc254

    
           exportCommand: `poetry export --with-credentials --format ${DependenciesFile.PIP} --output ${DependenciesFile.PIP}`,

(Looking at the code there, this likely applies to pipenv too)

Reproduction Steps

import * as cdk from 'aws-cdk-lib';
import * as lambdaPython from '@aws-cdk/aws-lambda-python-alpha';

export const app = new cdk.App();
const stack = new cdk.Stack(app, 'Stack', {});

new lambdaPython.PythonFunction(stack, 'Func', {
  entry: 'entry',
  index: 'entry/__init__.py',
  runtime: cdk.aws_lambda.Runtime.PYTHON_3_9,
});

Directory tree of entry (before running anything):

entry
├── entry
│   └── __init__.py
├── poetry.lock
└── pyproject.toml

poetry.lock:

package = []

[metadata]
lock-version = "1.1"
python-versions = "*"
content-hash = "115cf985d932e9bf5f540555bbdd75decbb62cac81e399375fc19f6277f8c1d8"

[metadata.files]

pyproject.toml:

[tool.poetry]
name = "entry"
version = "0.1.0"
description = ""
authors = []

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

__init__.py: empty

What did you expect to happen?

Temporary files shouldn't be written to disk (or at least, not intermixed with real source). For instance, the generated requirements.txt could be placed into any other directory in the docker image and this probably won't occur.

What actually happened?

After running cdk synth, a (empty, in this case) requirements.txt file appears:

entry
├── entry
│   └── __init__.py
├── poetry.lock
├── pyproject.toml
└── requirements.txt

CDK CLI Version

2.14.0 (build 762d71b)

Framework Version

No response

Node.js Version

v14.17.5

OS

macOS

Language

Typescript

Language Version

TypeScript 4.1.6

Other information

No response

The text was updated successfully, but these errors were encountered:

corymhall · 2022-03-04T13:41:20Z

@huonw this might be a simple fix. Right now we run all our commands and then copy everything to the output directory. We might want to just copy first and then run the commands in the output directory.

I am unassigning and marking this issue as p2, which means that we are unable to work on this immediately, but would definitely welcome contributions.

We use +1s to help prioritize our work, and are happy to revaluate this issue based on community feedback. You can reach out to the cdk.dev community on Slack to solicit support for reprioritization.

gshpychka · 2022-07-13T10:34:15Z

@corymhall Would this be as simple as moving L98 to right after L93 here?

What would make sense as a test for this?

corymhall · 2022-07-15T13:12:21Z

@corymhall Would this be as simple as moving L98 to right after L93 here?

What would make sense as a test for this?

Yeah I think that would do it. I think we can update one of the integration tests to do an assertion that the function still executes successfully (see integ.bundling.ts as an example).

…21967) When performing function bundling sometimes the bundling steps will write files. Currently all these commands are run from the `entry` which is the actually source code location. This leads to contaminating the source code with bundling artifacts. This PR re-orders the bundling steps to first move the `entry` to the `outputDir` and then perform the bundling steps there. I've also updated all of the integration tests to use the new integration test framework and assertions. fixes #19231 ---- ### All Submissions: * [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions · 2022-09-09T00:25:56Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

…ws#21967) When performing function bundling sometimes the bundling steps will write files. Currently all these commands are run from the `entry` which is the actually source code location. This leads to contaminating the source code with bundling artifacts. This PR re-orders the bundling steps to first move the `entry` to the `outputDir` and then perform the bundling steps there. I've also updated all of the integration tests to use the new integration test framework and assertions. fixes aws#19231 ---- ### All Submissions: * [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

lrav35 · 2022-09-16T12:57:36Z

hello @corymhall, is this slated to be released soon? Not sure what your release cycle looks like.

huonw added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Mar 3, 2022

github-actions bot added the @aws-cdk/aws-lambda-python label Mar 3, 2022

github-actions bot assigned corymhall Mar 3, 2022

corymhall removed their assignment Mar 4, 2022

corymhall added p2 effort/small Small work item – less than a day of effort and removed needs-triage This issue or PR still needs to be triaged. labels Mar 4, 2022

corymhall mentioned this issue May 16, 2022

(lambda-python): requirements.txt created in the entry folder when using Poetry #20296

Closed

corymhall mentioned this issue Sep 7, 2022

@aws-cdk/aws-lambda-python-alpha: Bundling lambdas with Poetry is broken #21867

Closed

corymhall self-assigned this Sep 8, 2022

corymhall added the in-progress This issue is being actively worked on. label Sep 8, 2022

corymhall mentioned this issue Sep 8, 2022

fix(lambda-python): bundling artifacts are written to the entry path #21967

Merged

4 tasks

mergify bot closed this as completed in #21967 Sep 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(aws-lambda-python): Using poetry requirements results in requirements.txt written to 'entry' directory #19231

(aws-lambda-python): Using poetry requirements results in requirements.txt written to 'entry' directory #19231

huonw commented Mar 3, 2022 •

edited

Loading

corymhall commented Mar 4, 2022

gshpychka commented Jul 13, 2022

corymhall commented Jul 15, 2022

github-actions bot commented Sep 9, 2022

lrav35 commented Sep 16, 2022

(aws-lambda-python): Using poetry requirements results in requirements.txt written to 'entry' directory #19231

(aws-lambda-python): Using poetry requirements results in requirements.txt written to 'entry' directory #19231

Comments

huonw commented Mar 3, 2022 • edited Loading

What is the problem?

Reproduction Steps

What did you expect to happen?

What actually happened?

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

corymhall commented Mar 4, 2022

gshpychka commented Jul 13, 2022

corymhall commented Jul 15, 2022

github-actions bot commented Sep 9, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

lrav35 commented Sep 16, 2022

huonw commented Mar 3, 2022 •

edited

Loading