chore(cli): prevent test interference #32270

rix0rrr · 2024-11-25T11:25:28Z

Our CLI unit tests were interfering with each other because they were writing files from and to the current directory, which is shared between all of them.

Solve it by making a non-writeable directory before running the tests, so that the tests that do that start throwing errors and we can identify them. Then fix those.

I tried papering over the issue by trying to create tempdirs automatically, but that started to introduce all kinds of errors in tests that were already doing tempdir management themselves, and it took too long to go and figure out what was wrong there, so I'm doing this instead.

Also in this PR:

Switch off global silent mode for the tests. It's very annoying if console.log statements don't make it out when you're debugging.
This PR caused a couple of lines in init.ts to be useless (they were there for tests), and removing those causes a lack of coverage in init.ts's entirety, so add some tests for that file.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Our CLI unit tests were interfering with each other because they were writing files from and to the current directory, which is shared between all of them. Solve it by making a non-writeable directory before running the tests, so that the tests that do that start throwing errors and we can identify them. Then fix those.

codecov · 2024-11-25T11:37:39Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.46%. Comparing base (421d327) to head (a534e33).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #32270      +/-   ##
==========================================
+ Coverage   77.17%   77.46%   +0.28%     
==========================================
  Files         105      105              
  Lines        7169     7168       -1     
  Branches     1315     1314       -1     
==========================================
+ Hits         5533     5553      +20     
+ Misses       1455     1433      -22     
- Partials      181      182       +1

Flag	Coverage Δ
suite.unit	`77.46% <100.00%> (+0.28%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
packages/aws-cdk	`77.46% <100.00%> (+0.28%)`	⬆️

---- 🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

iliapolo · 2024-11-25T12:13:30Z

packages/aws-cdk/test/jest-setup-after-env.ts

+
+beforeAll(() => {
+  tmpDir = path.join(os.tmpdir(), 'cdk-nonwritable');
+  if (!fs.existsSync(tmpDir)) {


So If I manually change the permissions of this directory back to writable - suddenly I can locally write tests that write to this directory.

Can we make this unique per execution?

This is not a security measure that we have to make sure nobody can circumvent.

It's a measure to help you prevent mistakes.

Also, the test will still fail on the build server (unless you put a chmod in your code, but again why are you bypassing a measure that is helping you write more robust tests? This is not for security purposes!)

Im not trying to tighten security, i'm trying to minimize scenarios where tests pass locally but fail on the server.

A contributor gets a weird permission error -> they don't know whats going on -> so they manually "fix" the issue by changing perms on the static directory. They write tests locally, all is well. But then, as you said, it fails on the build server. Thats not a great experience. What's the downside of having a unique dir per execution?

If someone gets as far as chmodding a directory that's called cdk-nonwritable... 🙄 I don't know how to help. I can name it cdk-nonwritable-on-purpose-do-not-chmod, but I don't know if that makes it better.

What I can do is re-chmod on every run. That doesn't require a new directory and also solves the problem where a developer will actively shoot themselves in the foot

If they are able to get to that directory, it seems they could grep the codebase to get to this specific file and read the comment there as well.

If there's a unique directory per execution, whatever manual permission change they do won't solve the problem.

Because they changed permission of /tmp/unique1/cdk-nonwritable, but now the error is coming from /tmp/unique2/cdk-nowritable

And they will keep getting the error until they understand how to properly solve it. Not critical, take it or leave it.

I did it another way

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

iliapolo

Remove the do-not-merge label at will.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

iliapolo · 2024-11-25T13:43:24Z

packages/aws-cdk/test/jest-setup-after-env.ts

+beforeAll(() => {
+  tmpDir = path.join(os.tmpdir(), 'cdk-nonwritable-on-purpose');
+  fs.mkdirSync(tmpDir, { recursive: true });
+  fs.chmodSync(tmpDir, 0o500);


That works too 👍

mergify · 2024-11-25T14:06:24Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2024-11-25T14:10:49Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2024-11-25T14:47:14Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

…nto huijbers/test-interference

aws-cdk-automation · 2024-11-25T15:26:04Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: a534e33
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2024-11-25T15:26:30Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

github-actions · 2024-11-25T15:26:56Z

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

rix0rrr requested a review from a team as a code owner November 25, 2024 11:25

aws-cdk-automation requested a review from a team November 25, 2024 11:25

github-actions bot added the p2 label Nov 25, 2024

rix0rrr requested a deployment to test-pipeline November 25, 2024 11:26 — with GitHub Actions Waiting

mergify bot added the contribution/core This is a PR that came from AWS. label Nov 25, 2024

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Nov 25, 2024

iliapolo reviewed Nov 25, 2024

View reviewed changes

This is no longer necessary, and uncovered

4a72ac0

rix0rrr requested a deployment to test-pipeline November 25, 2024 12:55 — with GitHub Actions Waiting

aws-cdk-automation previously requested changes Nov 25, 2024

View reviewed changes

aws-cdk-automation added the pr/needs-cli-test-run This PR needs CLI tests run against it. label Nov 25, 2024

Add some coverage

88d06de

rix0rrr requested a deployment to test-pipeline November 25, 2024 13:25 — with GitHub Actions Waiting

Add tests

910505a

rix0rrr requested a deployment to test-pipeline November 25, 2024 13:28 — with GitHub Actions Waiting

rix0rrr added 2 commits November 25, 2024 14:35

Always chmod directory so developers can't locally chmod it back

0c32162

Rename directory

e04f71e

rix0rrr requested a deployment to test-pipeline November 25, 2024 13:36 — with GitHub Actions Waiting

iliapolo added the pr/do-not-merge This PR should not be merged at this time. label Nov 25, 2024

iliapolo approved these changes Nov 25, 2024

View reviewed changes

rix0rrr added pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested and removed pr/do-not-merge This PR should not be merged at this time. labels Nov 25, 2024

aws-cdk-automation removed the pr/needs-cli-test-run This PR needs CLI tests run against it. label Nov 25, 2024

iliapolo reviewed Nov 25, 2024

View reviewed changes

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Nov 25, 2024

Run afterAll in reverse order

58b9c2f

rix0rrr requested a deployment to test-pipeline November 25, 2024 14:17 — with GitHub Actions Waiting

Merge branch 'main' into huijbers/test-interference

acfcfdc

rix0rrr requested a deployment to test-pipeline November 25, 2024 14:48 — with GitHub Actions Waiting

rix0rrr added 2 commits November 25, 2024 15:51

Cover that one line

415b32a

Merge branch 'huijbers/test-interference' of github.com:aws/aws-cdk i…

a534e33

…nto huijbers/test-interference

rix0rrr requested a deployment to test-pipeline November 25, 2024 14:56 — with GitHub Actions Waiting

mergify bot merged commit c5f895e into main Nov 25, 2024
16 of 17 checks passed

mergify bot deleted the huijbers/test-interference branch November 25, 2024 15:26

github-actions bot locked as resolved and limited conversation to collaborators Nov 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(cli): prevent test interference #32270

chore(cli): prevent test interference #32270

rix0rrr commented Nov 25, 2024 •

edited

Loading

codecov bot commented Nov 25, 2024 •

edited

Loading

iliapolo Nov 25, 2024

rix0rrr Nov 25, 2024

iliapolo Nov 25, 2024

rix0rrr Nov 25, 2024 •

edited

Loading

rix0rrr Nov 25, 2024

iliapolo Nov 25, 2024

rix0rrr Nov 25, 2024

aws-cdk-automation left a comment

iliapolo left a comment

iliapolo Nov 25, 2024

mergify bot commented Nov 25, 2024

mergify bot commented Nov 25, 2024

mergify bot commented Nov 25, 2024

aws-cdk-automation commented Nov 25, 2024

mergify bot commented Nov 25, 2024

github-actions bot commented Nov 25, 2024

chore(cli): prevent test interference #32270

chore(cli): prevent test interference #32270

Conversation

rix0rrr commented Nov 25, 2024 • edited Loading

codecov bot commented Nov 25, 2024 • edited Loading

Codecov Report

iliapolo Nov 25, 2024

Choose a reason for hiding this comment

rix0rrr Nov 25, 2024

Choose a reason for hiding this comment

iliapolo Nov 25, 2024

Choose a reason for hiding this comment

rix0rrr Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

rix0rrr Nov 25, 2024

Choose a reason for hiding this comment

iliapolo Nov 25, 2024

Choose a reason for hiding this comment

rix0rrr Nov 25, 2024

Choose a reason for hiding this comment

aws-cdk-automation left a comment

Choose a reason for hiding this comment

iliapolo left a comment

Choose a reason for hiding this comment

iliapolo Nov 25, 2024

Choose a reason for hiding this comment

mergify bot commented Nov 25, 2024

mergify bot commented Nov 25, 2024

mergify bot commented Nov 25, 2024

aws-cdk-automation commented Nov 25, 2024

AWS CodeBuild CI Report

mergify bot commented Nov 25, 2024

github-actions bot commented Nov 25, 2024

rix0rrr commented Nov 25, 2024 •

edited

Loading

codecov bot commented Nov 25, 2024 •

edited

Loading

rix0rrr Nov 25, 2024 •

edited

Loading