-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
@aws-cdk/aws-lambda-python-alpha: Bundling lambdas with Poetry is broken #21867
@aws-cdk/aws-lambda-python-alpha: Bundling lambdas with Poetry is broken #21867
Comments
It looks like the image doesn't have write access to I think the solution here is to do what we have been doing for the The new DockerFile should probably look something like: # The correct AWS SAM build image based on the runtime of the function will be
# passed as build arg. The default allows to do `docker build .` when testing.
ARG IMAGE=public.ecr.aws/sam/build-python3.7
FROM $IMAGE
ARG PIP_INDEX_URL
ARG PIP_EXTRA_INDEX_URL
ARG HTTPS_PROXY
# Ensure all users can write to pip cache
RUN mkdir /tmp/pip-cache && \
chmod -R 777 /tmp/pip-cache
ENV PIP_CACHE_DIR=/tmp/pip-cache
# Upgrade pip (required by cryptography v3.4 and above, which is a dependency of poetry)
RUN pip install --upgrade pip
# pipenv 2022.4.8 is the last version with Python 3.6 support
RUN pip install pipenv==2022.4.8 poetry
# Ensure all users can write to poetry cache
RUN mkdir /tmp/poetry-cache && \
chmod -R 777 /tmp/poetry-cache && \
poetry config cache-dir /tmp/poetry-cache
# create non root user and change allow execute command for non root user
RUN /sbin/useradd -u 1000 user && chmod 711 /
CMD [ "python" ] |
Until this is fixed, the workaround is to change the cache directory. const f = new PythonFunction(this, 'Lambda', {
runtime: Runtime.PYTHON_3_9,
entry: './lambda',
bundling: {
environment: { POETRY_VIRTUALENVS_IN_PROJECT: 'true' },
},
}) |
That workaround seems to work, thanks for the quick response! |
When I try to use that change-cache-location workaround, I get an error that looks like it's seeking a python binary which is not being sent to the built assets directory Here's my slightly cleaned up verbose output: Reading existing template for stack my-existing-stack.
my-existing-stack: deploying...
Waiting for stack CDKToolkit to finish creating or updating...
Preparing asset [ASSET_HASH]: {"path":"asset.[ASSET_HASH]","id":"[ASSET_HASH]","packaging":"zip","sourceHash":"[ASSET_HASH]","s3BucketParameter":"[AssetParametersASSET_HASH]S3Bucket81578626","s3KeyParameter":"[AssetParametersASSET_HASH]S3VersionKey3ED99592","artifactHashParameter":"[AssetParametersASSET_HASH]ArtifactHash7C848A52"}
Storing asset asset.[ASSET_HASH] at s3://[CDK_S3_STORAGE]/assets/[ASSET_HASH].zip
my-existing-stack: checking if we can skip deploy
my-existing-stack: template has changed
my-existing-stack: deploying...
[0%] start: Publishing [ASSET_HASH]:current
[0%] check: Check s3://[CDK_S3_STORAGE]/assets/[ASSET_HASH].zip
[0%] build: Zip /opt/my-app/cdk/cdk.out/asset.[ASSET_HASH] -> cdk.out/.cache/[ASSET_HASH].zip
Error: ENOENT: no such file or directory, stat '/opt/my-app/cdk/cdk.out/asset.[ASSET_HASH]/.venv/bin/python' I tried using your Dockerfile example up above with multiple versions of the base image, but oddly it's still getting some specific permission denied errors. I believe it might be an issue with the Dockerfile where Modifying the Dockerfile to wait until after pip runs to then ❯ ls -lah cdk.out/asset.[SHA]/.venv/bin/python
lrwxr-xr-x 1 me wheel 23B Sep 2 16:32 cdk.out/asset.[SHA]/.venv/bin/python -> /var/lang/bin/python3.9 As I'm on OSX, there is no such path to |
Temporary workaround - see <aws/aws-cdk#21867 (comment)>.
PR with @corymhall's workaround & run with the issue mentioned by @ryanandonian. |
@ryanandonian We are experiencing the same issue that you have detailed. The symlink is being copied instead of the executable and then becomes invalid because of it's location. What process copies this symlink? |
This is a workaround to issue aws/aws-cdk#21867
Same issue here:
How are folks getting around this? |
Unfortunately, I am temporarily removing poetry from my project in hopes that I can get a deploy working. |
I was able to deploy to localstack by exporting to a requirements.txt file and removing the poetry.lock and pyproject.toml. |
And while doing so, did you not experience the invalid symlink referenced by @ryanandonian earlier? |
It looks like something was changed in the base image and there is no longer write access to the `/tmp` directory which causes bundling with poetry to fail (see linked issue). This PR updates the Dockerfile to create a new cache location for both `pip` and `poetry` and switches to using a virtualenv for python so that it is no longer using root. To test this I executed the `integ.function.poetry` integration test both before (to reproduce the error) and after the fix. I'm actually not sure why our integration tests didn't start failing in the pipeline. The only thing I can think of is that we are caching the docker images and it just hasn't pulled down a newer one that has this issue. fixes #21867 ---- ### All Submissions: * [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
No. I was able to deploy completely. |
@corymhall what mechanism is responsible for copying that python binary to the output asset? And is @mikelane 's outcome consistent with your understanding of how that copy works? |
Has anyone actually managed to deploy lambda with 2.41.0 version that includes the fix while still using Poetry? Or does it require removing Poetry configs like @mikelane seems to have fixed his case. We are still getting this #21867 (comment) so to me it seems that fix doesn't actually fix the Poetry deployments. 🤔 |
after cleaning up all building images (to fetch the new one) and updated cdk client I now get a new error still hinting that poetry bunding is NOT fixed:
|
In case there's another case of a dependency blocking everything, such as <aws/aws-cdk#21867>. (Not setting to 0 (unlimited) in case of any future bugs in Dependabot creating crazy amounts of PRs.)
@ingwinlu @jarikujansuu Our integration tests are not failing with that error message. Can someone provide an example that I can use to reproduce the error? |
@corymhall Example branch and run. |
I got into a weird state with this one. I used my example project from the first post in the issue, and after upgrading to 2.41.0 I was able to Next, I added the So then I removed the bundling workaround again, and the deploy was still broken 🤔 This caused a lot of head-scratching, but I eventually figured out that the broken run with the workaround in place had created a Out of curiosity, I tried Cory's suggestion of changing the TL;DR: if you ever tried the |
@darylweir thanks for the analysis! Can still having the issue try what @darylweir has suggested and let me know if you are still having any issues? |
@corydozen I still need to look into the precise issue that is going on but I can +1 the behaviour to what @darylweir explained above. |
@darylweir @corymhall I apologize as this may be a dumb question - How are you folks editing the library itself? I see how that update would fix the issue we are having, but I'm unsure how we would edit that file and use the edited version. |
@darylweir I'm not familiar enough with the CDK output to detect what you mean, are you referring to the "Digest: sha256:e49b1f5fbe105cf14ed831ba8008d5f67ba1b6b2b68936b5d6b8093a88561e23" line? At least for |
@maj5004 Well, I did it in a hacky way: I just manually edited I guess the actual instructions are these ones |
@corymhall after removing previous hack |
@l0b0 it looks like your https://github.com/linz/geostore/blob/7514cd850a2236df27ca60afd340c5733680eafc/poetry.lock#L93 |
i executed the following steps to "cleanup" the aftermath:
|
@ingwinlu it may be an issue with a previously uploaded bad asset. If the bundling is fixed, but the asset hash hasn't changed from when the bundling was broken it won't upload the new asset (since it thinks it still exists). Can you see whether the asset hash has changed between deployments? |
aaah I forgot the assets s3 bucket. that would explain some of the "weird" behavior I noticed where things did not run the way I expected. instead of comparing hashes I whacked the bucket with an empty operation and that seems to have done it. All stacks could be deployed again. big thanks to all involved for fast responses + helpful comments even though I barely provided any data to go on. |
Following your steps listed there @ingwinlu , coupled with upgrading the version to Thanks to everyone who worked on getting this fix out quickly! |
|
In case there's another case of a dependency blocking everything, such as <aws/aws-cdk#21867>. (Not setting to 0 (unlimited) in case of any future bugs in Dependabot creating crazy amounts of PRs.) Co-authored-by: Jim Gan <[email protected]> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
aws/aws-cdk#21867 and aws/aws-cdk#21902 (jsii 1.69+ required)
Describe the bug
Our CI started breaking yesterday when trying to synthesise stacks using the
PythonFunction
construct. I was able to reproduce the failure locally using bothaws-lambda-python-alpha
in v2 andaws-lambda-python
in v1.The failure occurs when trying to setup the virtual env inside the bundling docker image. I assume something has changed in the latest
aws/sam/build-python3.9
docker image that breaks the bundling command.EDIT: we were using Python 3.9 lambdas, but I tested and builds also fail for 3.7 and 3.8 runtimes.
Expected Behavior
cdk synth
should succeed in bundling the lambda function.Current Behavior
The lambda bundling fails with an error from virtualenv:
virtualenv: error: argument dest: the destination . is not write-able at /
Full error output
Reproduction Steps
lib/cdktestv2-stack.ts
:bin/cdktestb2.ts
:lambda/pyproject.toml
:lambda/index.py
:and
poetry install
inlambda
to create a lock file.Then run
cdk synth
and watch it fall over.Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.39.0
Framework Version
No response
Node.js Version
16.14.0
OS
MacOS 10.15.7
Language
Typescript, Python
Language Version
Typescript 3.9.7, Python 3.9.1
Other information
No response
The text was updated successfully, but these errors were encountered: