Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3: autoDeleteObjects lambda chokes/hangs on large buckets #30573

Closed
isker opened this issue Jun 17, 2024 · 4 comments · Fixed by #30209 · 4 remaining pull requests
Closed

s3: autoDeleteObjects lambda chokes/hangs on large buckets #30573

isker opened this issue Jun 17, 2024 · 4 comments · Fixed by #30209 · 4 remaining pull requests
Labels
@aws-cdk/aws-s3 Related to Amazon S3 bug This issue is a bug. effort/medium Medium work item – several days of effort p1

Comments

@isker
Copy link
Contributor

isker commented Jun 17, 2024

Describe the bug

I recently had the mispleasure of trying to empty a bucket with ~600000 objects using CDK's autoDeleteObjects feature. What I observed was that each lambda invocation would get through a few tens of thousands of objects in relatively good time (a few minutes), then the lambda would grind to a halt doing very little until it reached its 15 minute timeout. This process then repeats with subsequent invocations of the lambda. I had to empty the bucket in the web console to make real progress toward deleting the bucket.

Expected Behavior

Lambda does not hang.

Current Behavior

Lambda hangs.

Reproduction Steps

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from 'aws-cdk-lib/aws-s3';

export class CdkTestStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'Bucket', {
      bucketName: 'isker-test',
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    })
  }
}

Load the resulting bucket with 100000 objects. Destroy the stack, and observe the lambda hanging until it hits its 15 minute timeout.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.124

Framework Version

No response

Node.js Version

20.11.1

OS

macOS

Language

TypeScript

Language Version

No response

Other information

I am filing this issue retroactively in order that my fix for it be merged. #30209

@isker isker added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 17, 2024
@github-actions github-actions bot added the @aws-cdk/aws-s3 Related to Amazon S3 label Jun 17, 2024
@khushail
Copy link
Contributor

thanks @isker for reporting this and submitting a PR!

@khushail khushail added p2 effort/medium Medium work item – several days of effort p1 and removed p2 needs-triage This issue or PR still needs to be triaged. labels Jun 17, 2024
@mergify mergify bot closed this as completed in #30209 Jun 17, 2024
@mergify mergify bot closed this as completed in 25835e4 Jun 17, 2024
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

1 similar comment
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

sarangarav pushed a commit to sarangarav/aws-cdk that referenced this issue Jun 21, 2024
…jects (aws#30209)

Fixes aws#30573.

I recently had the mispleasure of trying to empty a bucket with ~600000 objects using CDK's `autoDeleteObjects` feature. What I observed was that each lambda invocation would get through a few tens of thousands of objects in relatively good time (a few minutes), then the lambda would grind to a halt doing very little until it reached its 15 minute timeout. This process then repeats with subsequent invocations of the lambda.  I had to empty the bucket in the web console to make real progress toward deleting the bucket.

I have proven that the low memory allocated to the lambda (the default 128mb) plus this recursion is to blame. There is no need to recurse, and doing so will put pressure on the stack, the heap, and (because this is an async function) the event loop.

Switch the recursion to iteration.

aws#30209 (comment)

- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
mazyu36 pushed a commit to mazyu36/aws-cdk that referenced this issue Jun 22, 2024
…jects (aws#30209)

### Issue # (if applicable)
Fixes aws#30573.

### Reason for this change

I recently had the mispleasure of trying to empty a bucket with ~600000 objects using CDK's `autoDeleteObjects` feature. What I observed was that each lambda invocation would get through a few tens of thousands of objects in relatively good time (a few minutes), then the lambda would grind to a halt doing very little until it reached its 15 minute timeout. This process then repeats with subsequent invocations of the lambda.  I had to empty the bucket in the web console to make real progress toward deleting the bucket.

I have proven that the low memory allocated to the lambda (the default 128mb) plus this recursion is to blame. There is no need to recurse, and doing so will put pressure on the stack, the heap, and (because this is an async function) the event loop.

### Description of changes

Switch the recursion to iteration.

### Description of how you validated changes

aws#30209 (comment)

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@aws-cdk-automation
Copy link
Collaborator

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

@aws aws locked as resolved and limited conversation to collaborators Jul 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.