Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(cli): prevent test interference #32270

Merged
merged 10 commits into from
Nov 25, 2024
3 changes: 1 addition & 2 deletions packages/aws-cdk/jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,5 @@ module.exports = {

// We have many tests here that commonly time out
testTimeout: 30_000,
// These tests are too chatty. Shush.
silent: true,
setupFilesAfterEnv: ["<rootDir>/test/jest-setup-after-env.ts"],
};
18 changes: 18 additions & 0 deletions packages/aws-cdk/test/diff.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { Writable } from 'stream';
import { StringDecoder } from 'string_decoder';
import * as cxschema from '@aws-cdk/cloud-assembly-schema';
Expand All @@ -12,6 +14,22 @@ import { CdkToolkit } from '../lib/cdk-toolkit';
let cloudExecutable: MockCloudExecutable;
let cloudFormation: jest.Mocked<Deployments>;
let toolkit: CdkToolkit;
let oldDir: string;
let tmpDir: string;

beforeAll(() => {
// The toolkit writes and checks for temporary files in the current directory,
// so run these tests in a tempdir so they don't interfere with each other
// and other tests.
oldDir = process.cwd();
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'aws-cdk-test'));
process.chdir(tmpDir);
});

afterAll(() => {
process.chdir(oldDir);
fs.rmSync(tmpDir, { recursive: true, force: true });
});

describe('fixed template', () => {
const templatePath = 'oldTemplate.json';
Expand Down
64 changes: 64 additions & 0 deletions packages/aws-cdk/test/jest-setup-after-env.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';

/**
* Global test setup for Jest tests
*
* It's easy to accidentally write tests that interfere with each other by
* writing files to disk in the "current directory". To prevent this, the global
* test setup creates a directory in the temporary directory and chmods it to
* being non-writable. That way, whenever a test tries to write to the current
* directory, it will produce an error and we'll be able to find and fix the
* test.
*
* If you see `EACCES: permission denied`, you have a test that creates files
* in the current directory, and you should be sure to do it in a temporary
* directory that you clean up afterwards.
*
* ## Alternate approach
*
* I tried an approach where I would automatically try to create and clean up
* temp directories for every test, but it was introducing too many conflicts
* with existing test behavior (around specific ordering of temp directory
* creation and cleanup tasks that are already present) in many places that I
* didn't want to go and chase down.
*
*/

let tmpDir: string;
let oldDir: string;

beforeAll(() => {
tmpDir = path.join(os.tmpdir(), 'cdk-nonwritable');
if (!fs.existsSync(tmpDir)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So If I manually change the permissions of this directory back to writable - suddenly I can locally write tests that write to this directory.

Can we make this unique per execution?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a security measure that we have to make sure nobody can circumvent.

It's a measure to help you prevent mistakes.

Also, the test will still fail on the build server (unless you put a chmod in your code, but again why are you bypassing a measure that is helping you write more robust tests? This is not for security purposes!)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im not trying to tighten security, i'm trying to minimize scenarios where tests pass locally but fail on the server.

A contributor gets a weird permission error -> they don't know whats going on -> so they manually "fix" the issue by changing perms on the static directory. They write tests locally, all is well. But then, as you said, it fails on the build server. Thats not a great experience. What's the downside of having a unique dir per execution?

Copy link
Contributor Author

@rix0rrr rix0rrr Nov 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If someone gets as far as chmodding a directory that's called cdk-nonwritable... 🙄 I don't know how to help. I can name it cdk-nonwritable-on-purpose-do-not-chmod, but I don't know if that makes it better.

What I can do is re-chmod on every run. That doesn't require a new directory and also solves the problem where a developer will actively shoot themselves in the foot

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they are able to get to that directory, it seems they could grep the codebase to get to this specific file and read the comment there as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there's a unique directory per execution, whatever manual permission change they do won't solve the problem.

Because they changed permission of /tmp/unique1/cdk-nonwritable, but now the error is coming from /tmp/unique2/cdk-nowritable

And they will keep getting the error until they understand how to properly solve it. Not critical, take it or leave it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did it another way

fs.mkdirSync(tmpDir);
fs.chmodSync(tmpDir, 0o500);
}
oldDir = process.cwd();
process.chdir(tmpDir);
tmpDir = process.cwd(); // This will have resolved symlinks
});

/**
* We need a cleanup here
*
* 99% of the time, Jest runs the tests in a subprocess and this isn't
* necessary because we would have `chdir`ed in the subprocess.
*
* But sometimes we ask Jest with `-i` to run the tests in the main process,
* or if you only ask for a single test suite Jest runs the tests in the main
* process, and then we `chdir`ed the main process away.
*
* Jest will then try to write the `coverage` directory to the readonly directory,
* and fail. Chdir back to the original dir.
*
* Only if we are still in the tempdir, because if not then some other temporary
* directory cleanup block has already done the same and we shouldn't interfere
* with that.
*/
afterAll(() => {
if (process.cwd() === tmpDir) {
process.chdir(oldDir);
}
});
23 changes: 10 additions & 13 deletions packages/aws-cdk/test/notices.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -455,23 +455,20 @@ describe(CachedDataSource, () => {
});

test('retrieves data from the delegate when the file cannot be read', async () => {
const debugSpy = jest.spyOn(logging, 'debug');
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cdk-test'));
try {
const debugSpy = jest.spyOn(logging, 'debug');

if (fs.existsSync('does-not-exist.json')) {
fs.unlinkSync('does-not-exist.json');
}

const dataSource = dataSourceWithDelegateReturning(freshData, 'does-not-exist.json');
const dataSource = dataSourceWithDelegateReturning(freshData, `${tmpDir}/does-not-exist.json`);

const notices = await dataSource.fetch();

expect(notices).toEqual(freshData);
expect(debugSpy).not.toHaveBeenCalled();
const notices = await dataSource.fetch();

debugSpy.mockRestore();
expect(notices).toEqual(freshData);
expect(debugSpy).not.toHaveBeenCalled();

if (fs.existsSync('does-not-exist.json')) {
fs.unlinkSync('does-not-exist.json');
debugSpy.mockRestore();
} finally {
fs.rmSync(tmpDir, { recursive: true, force: true });
}
});

Expand Down
Loading