Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking: [CLI] testing posture improvements | Phase 1 #31929

Open
1 of 2 tasks
mrgrain opened this issue Oct 29, 2024 · 1 comment
Open
1 of 2 tasks

Tracking: [CLI] testing posture improvements | Phase 1 #31929

mrgrain opened this issue Oct 29, 2024 · 1 comment
Assignees
Labels
@aws-cdk/core Related to core CDK functionality cli Issues related to the CDK CLI management/tracking Issues that track a subject or multiple issues p1

Comments

@mrgrain
Copy link
Contributor

mrgrain commented Oct 29, 2024

Overview

As part of the build and release process, we are running a set of CLI integration tests if any of the CLI code has changed.
These tests currently take 50+ min each and are frequently unstable. We are also missing unit tests where we could have them. In general our branches coverage percentages are comparatively low (68% as of 2024-10-29).

Implementation

We are addressing the shortfall through a number of improvements. See issues for details.

Issue list

@mrgrain mrgrain added management/tracking Issues that track a subject or multiple issues needs-triage This issue or PR still needs to be triaged. p1 @aws-cdk/core Related to core CDK functionality and removed needs-triage This issue or PR still needs to be triaged. labels Oct 29, 2024
@mrgrain mrgrain changed the title Tracking: CLI Integration Test improvements Tracking: CLI Test posture improvements Oct 29, 2024
@mrgrain mrgrain changed the title Tracking: CLI Test posture improvements Tracking: CLI testing posture improvements Oct 29, 2024
@mrgrain mrgrain added cli Issues related to the CDK CLI p1 and removed p1 labels Oct 29, 2024
@iliapolo
Copy link
Contributor

iliapolo commented Nov 6, 2024

Some additional thoughts on this one:

  • Add coverage reports based on integration tests.
  • Run all integ tests in a cross account scenario as well.
  • Run all integ tests in a multi environment scenario as well. E.g deploy the same stack to multiple envs within a single CLI execution. This will catch any code that incorrectly assumes a single environment (bootstrap version checks for example).
  • IAM regression testing. Ensure commands don't introduce new role assumptions or new actions because users may have configured locked down permissions. This would have prevented ❗ NOTICE: cdk diff tries to assume 'deploy' role and fails to authorize. #29483. We should recognize though that sometimes we would like to rollout changes like these so we need a bypass mechanism as well. Catching these in regression gives us a chance to properly communicate these changes even though we sometimes bypass the validation.
    • We should be able to automatically capture all roles and actions performed by CLI and write them as a snapshot.

@iliapolo iliapolo changed the title Tracking: CLI testing posture improvements Tracking: CLI testing posture improvements | Phase 1 Nov 18, 2024
@iliapolo iliapolo changed the title Tracking: CLI testing posture improvements | Phase 1 Tracking: CLI testing posture improvements Nov 18, 2024
@iliapolo iliapolo changed the title Tracking: CLI testing posture improvements Tracking: CLI testing posture improvements | Phase 1 Nov 18, 2024
@iliapolo iliapolo changed the title Tracking: CLI testing posture improvements | Phase 1 Tracking: [CLI] testing posture improvements | Phase 1 Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/core Related to core CDK functionality cli Issues related to the CDK CLI management/tracking Issues that track a subject or multiple issues p1
Projects
None yet
Development

No branches or pull requests

2 participants