Description
Please add your +1 👍 to let us know you have encountered this
Status: RESOLVED
Overview:
Release v2.163.0 introduced an issues with cross-account deployments. The AWS CDK fails with the error Need to perform AWS calls for account XXXXX, but the current credentials are for YYYYY. The root cause is a new check for the bootstrap version added in v2.163.0. The check needs to make an API call to AWS CloudFormation to describe the bootstrap stack. This API call was using incorrect credentials.
Complete Error Message:
Need to perform AWS calls for account XXXXXXXXXXXX, but the current credentials are for YYYYYYYYYYYY
Workaround:
Downgrade the AWS CDK CLI to version 2.162.1
Solution:
Upgrade the AWS CDK CLI to version 2.163.1
Related Issues:
n/a
Original issue:
Describe the bug
Release v2.163.0 causes deployments using cross account role assumption to fail with the error Need to perform *** calls for account XXXXX, but the current credentials are for YYYYY. This is presumably caused by #31623.
Based on comments on this PR (as it has no description or linked issue) this was done to address a vulnerability when the assets bucket for an account is not longer in the target account, but this failure is occurring when the assets bucket is still in the target account with no change.
These deployments and buckets are in the ap-southeast-2 region.
Regression Issue
Confirmed Regression
Last Known Working CDK Version
2.162.1
Expected Behavior
Cross account deployments should succeed unless the asset bucket is no longer in the target account.
Current Behavior
Cross account deployments fail with the error Need to perform *** calls for account XXXXX, but the current credentials are for YYYYY
cdk diff succeeds as expected, and cloudtrail shows role assumption is successful.
Debug log excerpt just before the failure:
[04:15:56] datadog-lambda-apm-test-dev: check: Check s3://cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2/2819175352ad1ce0dae768e83fc328fb70fb5f10b4a8ff0ccbcb791f02b0716d.zip
[04:15:56] [*** s3 200 0.06s 0 retries] listObjectsV2({
Bucket: 'cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2',
Prefix: 'fdfffca2098cb4f12a1dd3f0f2f98b4606063436cdfc311da0589251fa61cbe0.json',
MaxKeys: 1
})
[04:15:56] [*** s3 200 0.074s 0 retries] listObjectsV2({
Bucket: 'cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2',
Prefix: '7fbaff7dbc84f82c58c19512abc1c6e165bebea0038d20da2d2ee153808fee70.zip',
MaxKeys: 1
})
[04:15:56] datadog-lambda-apm-test-dev: found: Found s3://cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2/7fbaff7dbc84f82c58c19512abc1c6e165bebea0038d20da2d2ee153808fee70.zip
[04:15:56] [*** s3 200 0.06s 0 retries] listObjectsV2({
Bucket: 'cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2',
Prefix: '2819175352ad1ce0dae768e83fc328fb70fb5f10b4a8ff0ccbcb791f02b0716d.zip',
MaxKeys: 1
})
[04:15:56] datadog-lambda-apm-test-dev: found: Found s3://cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2/2819175352ad1ce0dae768e83fc328fb70fb5f10b4a8ff0ccbcb791f02b0716d.zip
[04:15:56] 3 total assets, 1 still need to be published
[04:15:56] [trace] SdkProvider#resolveEnvironment()
[04:15:56] [trace] SdkProvider#baseCredentialsPartition()
[04:15:56] [trace] SdkProvider#resolveEnvironment()
[04:15:56] [trace] SdkProvider#obtainBaseCredentials()
[04:15:56] [trace] SdkProvider#defaultAccount()
[04:15:56] [trace] SdkProvider#defaultCredentials()
[04:15:56] [trace] SDK#currentAccount()
[04:15:56] [trace] SDK#forceCredentialRetrieval()
[04:15:56] Retrieved account ID YYYYYYYYYYYY from disk cache
[04:15:56] [trace] SDK#ssm()
[04:15:56] [trace] SDK#wrapServiceErrorHandling()
[04:15:56] [*** ssm 200 0.062s 0 retries] getParameter({ Name: '/cdk-bootstrap/hnb659fds/version' })
datadog-lambda-apm-test-dev: start: Building fdfffca2098cb4f12a1dd3f0f2f98b4606063436cdfc311da0589251fa61cbe0:XXXXXXXXXXXX-ap-southeast-2
datadog-lambda-apm-test-dev: success: Built fdfffca2098cb4f12a1dd3f0f2f98b4606063436cdfc311da0589251fa61cbe0:XXXXXXXXXXXX-ap-southeast-2
[04:15:56] [trace] SdkProvider#resolveEnvironment()
[04:15:56] [trace] SdkProvider#baseCredentialsPartition()
[04:15:56] [trace] SdkProvider#resolveEnvironment()
[04:15:56] [trace] SdkProvider#obtainBaseCredentials()
[04:15:56] [trace] SdkProvider#defaultAccount()
[04:15:56] [trace] SdkProvider#defaultCredentials()
[04:15:56] [trace] SDK#currentAccount()
[04:15:56] [trace] SDK#forceCredentialRetrieval()
[04:15:56] Retrieved account ID YYYYYYYYYYYY from disk cache
[04:15:56] [trace] SdkProvider#forEnvironment()
[04:15:56] [trace] SdkProvider#resolveEnvironment()
[04:15:56] [trace] SdkProvider#obtainBaseCredentials()
[04:15:56] [trace] SdkProvider#defaultAccount()
[04:15:56] [trace] SdkProvider#defaultCredentials()
[04:15:56] Notices refreshed
Need to perform *** calls for account XXXXXXXXXXXX, but the current credentials are for YYYYYYYYYYYY
[04:15:56] Error: Need to perform *** calls for account XXXXXXXXXXXX, but the current credentials are for YYYYYYYYYYYY
at SdkProvider.forEnvironment (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/api/aws-auth/sdk-provider.ts:195:60)
at Deployments.cachedSdkForEnvironment (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/api/deployments.ts:918:17)
at Deployments.allowCrossAccountAssetPublishingForEnv (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/api/deployments.ts:863:20)
at Deployments.publishSingleAsset (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/api/deployments.ts:855:62)
at Object.publishAsset (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/cdk-toolkit.ts:254:7)
at /home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/util/work-graph.ts:111:11
Reproduction Steps
- Create a role in account A
- Bootstrap account B with account A (or the role in account A) as the trusted principle
- Assume role in account A
- Run
npx cdk deployon a stack with account B as the env.account property. May need to also set region to ap-southeast-2, I have not been able to test this in us-east-1 yet.
Possible Solution
Roll back #31623 until full integration testing can be performed.
Additional Information/Context
This change has caused all deployments from our CICD to fail, as they make use of cross account role assumption and use the latest CDK V2 CLI unless otherwise specified (due to caret version matching to avoid breaking changes). I would like to request that this change be rolled back until it can be thoroughly integration tested. If this is not possible, the new expected requirements for cross account CDK deployments need to be clearly documented.
CDK CLI Version
2.163.0
Framework Version
No response
Node.js Version
v20.13.1
OS
Debian Linux
Language
TypeScript
Language Version
TypeScript (5.6.3)
Other information
No response
Activity