Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): garbage collect ecr assets (under --unstable flag) #31841

Merged
merged 27 commits into from
Oct 26, 2024
Merged

Conversation

kaizencc
Copy link
Contributor

@kaizencc kaizencc commented Oct 21, 2024

Follow up to #31611 which introduced S3 Asset Garbage Collection

ECR Asset Garbage Collection

cdk gc now collects ECR assets.

cdk gc aws://0123456789012/us-east-1 \
  --unstable='gc' \
  --type='ecr'

or

cdk gc aws://0123456789012/us-east-1 \
  --unstable='gc' \
  --type='all'

all other options are duplicated from s3.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added the p2 label Oct 21, 2024
@aws-cdk-automation aws-cdk-automation requested a review from a team October 21, 2024 23:14
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Oct 21, 2024
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@aws-cdk-automation aws-cdk-automation added the pr/needs-cli-test-run This PR needs CLI tests run against it. label Oct 21, 2024

// We delete images that are not referenced in ActiveAssets and have the Isolated Tag with a date
// earlier than the current time - grace period.
deletables = isolated.filter(img => img.isolatedTagBefore(new Date(currentTime - (graceDays * DAY))));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do tags even work this way in an ECR repository?

Doesn't a tag only ever refer to exactly 1 image?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I think repos are by default created with tag immutability, because that's what security wants.

Copy link
Contributor

@rix0rrr rix0rrr Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the difference here is:

  • In S3, tags are nonfunctional metadata
  • In ECR (Docker), tags are a functional part of how you address the image

(They're not the same type of tag)

Copy link
Contributor

@rix0rrr rix0rrr Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can AWS-tag the repository itself... but it has limits key <128 chars, value <256 chars.

Don't know about the max amount of tags.

You could put the following tags on the repo:

aws-cdk:isolated:dde07f18495b1339c13905d5442a78c5 = 1729615771053
aws-cdk:isolated:3bbc165593b6e2a4dcd6e7d72ac51fac = 1729615783885

And never do more than X tags like this. We'll eventually clean out the repo, it just might take a while

Copy link
Contributor Author

@kaizencc kaizencc Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't a tag only ever refer to exactly 1 image?

you can have multiple tags per image.

the id of an image is its digest, which is different than its tag apparently.

Also, I think repos are by default created with tag immutability, because that's what security wants.

tag immutability shouldn't apply here - we are not overwriting any existing tags.

you could argue that it looks weird to tag our isolated images this way (because you can in theory pull from the isolated tag if you really wanted to) -- but i think this is at least doable. unless we really can't do this, i prefer tagging images this way as opposed to using the repo-level tags

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an example of what it looks like in the console:

Screenshot 2024-10-22 at 4 29 01 PM

Copy link
Contributor

@rix0rrr rix0rrr Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't a tag only ever refer to exactly 1 image?

you can have multiple tags per image.

That's not what I said though. Can you have 2 images with the same tag?

If not, how are you going to tag multiple images for garbage collection? Are you guaranteeing that the timestamps are all unique?

I guess conceivably it might be doable if you use JavaScript timestamps with millisecond resolution and increment them for each; we'd need to tag more than 1000 images/second for that to become inaccurate enough to care about it (and even then... meh).

Even given this... I'm not sure I'm comfortable putting this metadata in a load-bearing location. I guess we can try it and see if it breaks things, or puts undue load on ECR.

(A bit of a design overview in a doc comment of the class would have been extremely helpful reasoning through these concerns)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've decided offline to go with the Docker tags method as repository tags are limited in number (~50 metadata tags per resource). The issue with Docker tags is ensuring that each tag is unique, and we will do that via timestamps + increments.

@kaizencc kaizencc marked this pull request as ready for review October 22, 2024 23:22
@kaizencc kaizencc added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Oct 22, 2024
@kaizencc kaizencc requested a review from rix0rrr October 22, 2024 23:35
@@ -1,7 +1,7 @@
import { Tag } from '../../cdk-toolkit';

export const BUCKET_NAME_OUTPUT = 'BucketName';
export const REPOSITORY_NAME_OUTPUT = 'RepositoryName';
export const REPOSITORY_NAME_OUTPUT = 'ImageRepositoryName';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was wrong and was not used anywhere

@kaizencc kaizencc added pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested and removed pr/needs-cli-test-run This PR needs CLI tests run against it. labels Oct 23, 2024
@aws-cdk-automation aws-cdk-automation dismissed their stale review October 24, 2024 00:00

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Oct 24, 2024
Comment on lines +15 to +16
export const S3_ISOLATED_TAG = 'aws-cdk:isolated';
export const ECR_ISOLATED_TAG = 'aws-cdk.isolated'; // ':' is not valid in ECR tags
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@rix0rrr rix0rrr added the pr/do-not-merge This PR should not be merged at this time. label Oct 24, 2024
Copy link
Contributor

@rix0rrr rix0rrr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conditionally approved, modulo some tag handling.

const date = Date.now();
let imageTag;
try {
imageTag = `${ECR_ISOLATED_TAG}-${i}-${String(date)}`;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please turn these fields around, so that they more or less lexicographically sort correctly.

Also -- you have logic around building and splitting this string. Would be nice if that logic lived together. So the building logic should live next to the splitting logic, most likely on ImageAsset then.

@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Oct 24, 2024
@aws-cdk-automation
Copy link
Collaborator

➡️ PR build request submitted to test-main-pipeline ⬅️

A maintainer must now check the pipeline and add the pr-linter/cli-integ-tested label once the pipeline succeeds.

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 8e2d92d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@kaizencc kaizencc removed the pr/do-not-merge This PR should not be merged at this time. label Oct 26, 2024
Copy link
Contributor

mergify bot commented Oct 26, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit da85e54 into main Oct 26, 2024
15 checks passed
@mergify mergify bot deleted the conroy/ecrgc branch October 26, 2024 05:28
Copy link

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
contribution/core This is a PR that came from AWS. p2 pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested pr-linter/exempt-integ-test The PR linter will not require integ test changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants