-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(cli): prevent test interference #32270
Merged
Merged
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
f732c9e
chore(cli): prevent test interference
rix0rrr 4a72ac0
This is no longer necessary, and uncovered
rix0rrr 88d06de
Add some coverage
rix0rrr 910505a
Add tests
rix0rrr 0c32162
Always chmod directory so developers can't locally `chmod` it back
rix0rrr e04f71e
Rename directory
rix0rrr 58b9c2f
Run afterAll in reverse order
rix0rrr acfcfdc
Merge branch 'main' into huijbers/test-interference
rix0rrr 415b32a
Cover that one line
rix0rrr a534e33
Merge branch 'huijbers/test-interference' of github.com:aws/aws-cdk i…
rix0rrr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
import * as fs from 'fs'; | ||
import * as os from 'os'; | ||
import * as path from 'path'; | ||
|
||
/** | ||
* Global test setup for Jest tests | ||
* | ||
* It's easy to accidentally write tests that interfere with each other by | ||
* writing files to disk in the "current directory". To prevent this, the global | ||
* test setup creates a directory in the temporary directory and chmods it to | ||
* being non-writable. That way, whenever a test tries to write to the current | ||
* directory, it will produce an error and we'll be able to find and fix the | ||
* test. | ||
* | ||
* If you see `EACCES: permission denied`, you have a test that creates files | ||
* in the current directory, and you should be sure to do it in a temporary | ||
* directory that you clean up afterwards. | ||
* | ||
* ## Alternate approach | ||
* | ||
* I tried an approach where I would automatically try to create and clean up | ||
* temp directories for every test, but it was introducing too many conflicts | ||
* with existing test behavior (around specific ordering of temp directory | ||
* creation and cleanup tasks that are already present) in many places that I | ||
* didn't want to go and chase down. | ||
* | ||
*/ | ||
|
||
let tmpDir: string; | ||
let oldDir: string; | ||
|
||
beforeAll(() => { | ||
tmpDir = path.join(os.tmpdir(), 'cdk-nonwritable'); | ||
if (!fs.existsSync(tmpDir)) { | ||
fs.mkdirSync(tmpDir); | ||
fs.chmodSync(tmpDir, 0o500); | ||
} | ||
oldDir = process.cwd(); | ||
process.chdir(tmpDir); | ||
tmpDir = process.cwd(); // This will have resolved symlinks | ||
}); | ||
|
||
/** | ||
* We need a cleanup here | ||
* | ||
* 99% of the time, Jest runs the tests in a subprocess and this isn't | ||
* necessary because we would have `chdir`ed in the subprocess. | ||
* | ||
* But sometimes we ask Jest with `-i` to run the tests in the main process, | ||
* or if you only ask for a single test suite Jest runs the tests in the main | ||
* process, and then we `chdir`ed the main process away. | ||
* | ||
* Jest will then try to write the `coverage` directory to the readonly directory, | ||
* and fail. Chdir back to the original dir. | ||
* | ||
* Only if we are still in the tempdir, because if not then some other temporary | ||
* directory cleanup block has already done the same and we shouldn't interfere | ||
* with that. | ||
*/ | ||
afterAll(() => { | ||
if (process.cwd() === tmpDir) { | ||
process.chdir(oldDir); | ||
} | ||
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So If I manually change the permissions of this directory back to writable - suddenly I can locally write tests that write to this directory.
Can we make this unique per execution?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not a security measure that we have to make sure nobody can circumvent.
It's a measure to help you prevent mistakes.
Also, the test will still fail on the build server (unless you put a
chmod
in your code, but again why are you bypassing a measure that is helping you write more robust tests? This is not for security purposes!)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Im not trying to tighten security, i'm trying to minimize scenarios where tests pass locally but fail on the server.
A contributor gets a weird permission error -> they don't know whats going on -> so they manually "fix" the issue by changing perms on the static directory. They write tests locally, all is well. But then, as you said, it fails on the build server. Thats not a great experience. What's the downside of having a unique dir per execution?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If someone gets as far as
chmod
ding a directory that's calledcdk-nonwritable
... 🙄 I don't know how to help. I can name itcdk-nonwritable-on-purpose-do-not-chmod
, but I don't know if that makes it better.What I can do is re-
chmod
on every run. That doesn't require a new directory and also solves the problem where a developer will actively shoot themselves in the footThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If they are able to get to that directory, it seems they could
grep
the codebase to get to this specific file and read the comment there as well.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there's a unique directory per execution, whatever manual permission change they do won't solve the problem.
And they will keep getting the error until they understand how to properly solve it. Not critical, take it or leave it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did it another way