-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
s3: autoDeleteObjects
lambda chokes/hangs on large buckets
#30573
Comments
thanks @isker for reporting this and submitting a PR! |
|
1 similar comment
|
…jects (aws#30209) Fixes aws#30573. I recently had the mispleasure of trying to empty a bucket with ~600000 objects using CDK's `autoDeleteObjects` feature. What I observed was that each lambda invocation would get through a few tens of thousands of objects in relatively good time (a few minutes), then the lambda would grind to a halt doing very little until it reached its 15 minute timeout. This process then repeats with subsequent invocations of the lambda. I had to empty the bucket in the web console to make real progress toward deleting the bucket. I have proven that the low memory allocated to the lambda (the default 128mb) plus this recursion is to blame. There is no need to recurse, and doing so will put pressure on the stack, the heap, and (because this is an async function) the event loop. Switch the recursion to iteration. aws#30209 (comment) - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…jects (aws#30209) ### Issue # (if applicable) Fixes aws#30573. ### Reason for this change I recently had the mispleasure of trying to empty a bucket with ~600000 objects using CDK's `autoDeleteObjects` feature. What I observed was that each lambda invocation would get through a few tens of thousands of objects in relatively good time (a few minutes), then the lambda would grind to a halt doing very little until it reached its 15 minute timeout. This process then repeats with subsequent invocations of the lambda. I had to empty the bucket in the web console to make real progress toward deleting the bucket. I have proven that the low memory allocated to the lambda (the default 128mb) plus this recursion is to blame. There is no need to recurse, and doing so will put pressure on the stack, the heap, and (because this is an async function) the event loop. ### Description of changes Switch the recursion to iteration. ### Description of how you validated changes aws#30209 (comment) ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one. |
Describe the bug
I recently had the mispleasure of trying to empty a bucket with ~600000 objects using CDK's
autoDeleteObjects
feature. What I observed was that each lambda invocation would get through a few tens of thousands of objects in relatively good time (a few minutes), then the lambda would grind to a halt doing very little until it reached its 15 minute timeout. This process then repeats with subsequent invocations of the lambda. I had to empty the bucket in the web console to make real progress toward deleting the bucket.Expected Behavior
Lambda does not hang.
Current Behavior
Lambda hangs.
Reproduction Steps
Load the resulting bucket with 100000 objects. Destroy the stack, and observe the lambda hanging until it hits its 15 minute timeout.
Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.124
Framework Version
No response
Node.js Version
20.11.1
OS
macOS
Language
TypeScript
Language Version
No response
Other information
I am filing this issue retroactively in order that my fix for it be merged. #30209
The text was updated successfully, but these errors were encountered: