Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(aws-lambda-python): Asset bundling fails on Windows #18861

Closed
gshpychka opened this issue Feb 7, 2022 · 14 comments · Fixed by #19270
Closed

(aws-lambda-python): Asset bundling fails on Windows #18861

gshpychka opened this issue Feb 7, 2022 · 14 comments · Fixed by #19270
Labels
@aws-cdk/aws-lambda-python bug This issue is a bug. in-progress This issue is being actively worked on. os/windows Related specifically to Windows behavior p2

Comments

@gshpychka
Copy link
Contributor

gshpychka commented Feb 7, 2022 •
edited
Loading

What is the problem?

aws-lambda-python does not work, bundling fails.

Reproduction Steps

Use the PythonFunction construct. Peform a synth.

Code excerpt:

        lambda_python.PythonFunction(
            self,
            "my_func",
            entry=os.path.join("src", "my_lambda"),
            handler="handler",
            index="my_index.py",
            runtime=typing.cast(lambda_.Runtime, lambda_.Runtime.PYTHON_3_9)
        )

What did you expect to happen?

The lambda is bundled successfully.

What actually happened?

Error during synth:
cp: cannot copy a directory, '/asset_input/', into itself, 'asset-output'.
Error: Failed to bundle asset ...

CDK CLI Version

2.10.0

Framework Version

2.10.0

Node.js Version

16.13.0

OS

Windows

Language

Python

Language Version

No response

Other information

The same works on Linux somehow. This is related - #18301

Pinning aws-cdk.aws-lambda-python-alpha==2.3.0a0 solves the issue.
@gshpychka gshpychka added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 7, 2022
@NGL321 NGL321 added os/windows Related specifically to Windows behavior p2 and removed needs-triage This issue or PR still needs to be triaged. labels Feb 14, 2022
@corymhall
Copy link
Contributor

@gshpychka I'm having a hard time reproducing this since I don't have a windows machine. The weird part is that the behavior should not be any different between linux and windows because the bundling is happening in a linux docker container. Can you try reproducing this on a fresh project with just the minimum required dependencies?

@corymhall corymhall added needs-reproduction This issue needs reproduction. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Feb 16, 2022
@github-actions
Copy link

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Feb 18, 2022
@gbisaga
Copy link

gbisaga commented Feb 24, 2022

I am having the same problem with

"@aws-cdk/aws-lambda-python-alpha": "^2.13.0-alpha.0",
"aws-cdk-lib": "^2.13.0",

Here's a sample minimal project @corymhall :

https://github.com/gbisaga/aws-cdk-issues-18861

@victorwrotethis
Copy link

@gbisaga
It might be worth re-opening in another issue as the author's location likely means they are occupied with other things and leave them unable to respond and they don't seem to reopen it. It is how they addressed it the first time as well after #18301.

You can still bypass this issue by using v 2.3.0-alpha.0. I still got that working with v2.14 of the aws-cdk-lib without it introducing problems. But I would imagine it will on the long run.

@corymhall corymhall reopened this Mar 3, 2022
@corymhall
Copy link
Contributor

@gbisaga thanks for the sample project! This one really has me stumped. My hunch is that it is something with the way Docker handles volumes on Windows, but I might need some help running down a root cause.

@gbisaga
Copy link

gbisaga commented Mar 3, 2022

Of course! Please let me know if you would like me to try anything. I'm not that familiar with the internals of CDK, but I would love to learn more about it! Looking at the timing, I'm wondering if it has anything to do with the recent Docker Desktop changes. I have not experienced any other problems with it, but that might explain both why it doesn't have problems on linux and also why it just started happening now.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Mar 3, 2022
@corymhall corymhall mentioned this issue Mar 4, 2022
Closed
@corymhall
Copy link
Contributor

@gbisaga can you try running a test to see if it throws the same error?

docker run --rm -v /tmp/copy-test:/asset-input:delegated -v /tmp/copy-test-output:/asset-output:delegated -w /asset-input public.ecr.aws/sam/build-python3.9 cp -rT /asset-input/ /asset-output

You'll need to replace the volume source paths to windows paths so you'll end up with something like (depending on where you create the directories)

docker run --rm -v C:\\copy-test:/asset-input:delegated -v C:\\copy-test-output:/asset-output:delegated -w /asset-input public.ecr.aws/sam/build-python3.9 cp -rT /asset-input/ /asset-output

I created two directories /tmp/copy-test and /tmp/copy-test-output. I cloned your test repository into copy-test so I ended up with

/tmp/copy-test

/tmp/copy-test
└── aws-cdk-issues-18861

And after running the test I had

/tmp/copy-test-output
└── aws-cdk-issues-18861

@corymhall corymhall mentioned this issue Mar 4, 2022
Closed
@kichik
Copy link

kichik commented Mar 4, 2022 •
edited
Loading

I have also been having this issue for a while now. I used procmon.exe to get the Docker command that CDK runs and it's:

"C:\Program Files\Docker\Docker\resources\bin\com.docker.cli.exe" run --rm -u 1000:1000 -v C:\...\my-proj\my-lambda:/
asset-input:delegated -v C:\...\my-proj\cdk.out\asset.94afcae579c0c4746c303352f6aae43209f50ce029812eb3c005bdd3fb8519bb:/asset-output:delegated -w /asset-input cdk-6bbe7af8159970c0254f59e44d4d98e5cd784e14832228c913df8ea
651d8b90d bash -c "python -m pip install -r requirements.txt -t \asset-output && cp -rT /asset-input/ \asset-output"

The issue seems to be the last two \asset-output. Once I change those to /asset-output the command works again. I'm not sure why CDK chooses to use Windows style slashes inside Docker.

@corymhall corymhall mentioned this issue Mar 7, 2022
Merged
@NGL321 NGL321 added in-progress This issue is being actively worked on. and removed closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. needs-reproduction This issue needs reproduction. labels Mar 7, 2022
@mergify mergify bot closed this as completed in #19270 Mar 8, 2022
mergify bot pushed a commit that referenced this issue Mar 8, 2022
@corymhall
fix(lambda-python): asset bundling fails on windows (#19270)
0da57da 
The output directory should always use posix paths since this directory
is being used within a linux docker container.

I did not add any tests because it looks like there is no way to mock the `path` platform detection.

fixes #18861


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Mar 8, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@corymhall
Copy link
Contributor

Leaving this open until we can confirm the fix works.

@corymhall corymhall reopened this Mar 8, 2022
@kichik
Copy link

kichik commented Mar 8, 2022

Can I help confirm before the next release somehow?

TheRealAmazonKendra pushed a commit to TheRealAmazonKendra/aws-cdk that referenced this issue Mar 11, 2022
@corymhall @TheRealAmazonKendra
fix(lambda-python): asset bundling fails on windows (aws#19270)
66c5bea 
The output directory should always use posix paths since this directory
is being used within a linux docker container.

I did not add any tests because it looks like there is no way to mock the `path` platform detection.

fixes aws#18861


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@corymhall
Copy link
Contributor

@kichik can you try with the latest version? The fix should be in 1.148.0 and 2.17.0(just released).

@kichik
Copy link

kichik commented Mar 17, 2022

It works! Thanks! 🥳

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-lambda-python bug This issue is a bug. in-progress This issue is being actively worked on. os/windows Related specifically to Windows behavior p2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(lambda-python): asset bundling fails on windows
6 participants
@kichik @NGL321 @gbisaga @victorwrotethis @gshpychka @corymhall