-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(redshift): optionally reboot Clusters to apply parameter changes #22063
Merged
mergify
merged 25 commits into
aws:main
from
dontirun:feat-reboot-cluster-for-parameter-updates
Feb 17, 2023
Merged
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
8b8798c
inital implementation
dontirun 68ebe28
add lambda handler
dontirun f262a52
documentation, tests, and bugfixes
6c1122b
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
c61f994
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
6fe9387
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
63400f6
fix: allow for enabling the reboot feature before the parameter group…
a5d1ee5
test: comment out assertions
153345a
chore: commit deleted asstes
f31b6dd
chore: update snapshot
a01c334
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
52916e6
tests: regenerate snapshot
c445b0f
chore: commenting out diff assets in integ test
26fbbaa
docs: add testing procedure to integ test
b732dcf
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
692d1ab
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
762f2bf
test: update assertions on integ test
dae7883
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
dontirun 75ebd28
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
dontirun b6c323d
refactor: removing unknown-error from reboot actions
dontirun 1c69185
Merge branch 'master' into feat-reboot-cluster-for-parameter-updates
dontirun 80848a9
Update packages/@aws-cdk/aws-redshift/lib/cluster-parameter-change-re…
dontirun f91ffeb
refactor: use guard clause for clarity
a76b887
refactor: update guard clause
16ba85f
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
mergify[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
53 changes: 53 additions & 0 deletions
53
packages/@aws-cdk/aws-redshift/lib/cluster-parameter-change-reboot-handler/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
// eslint-disable-next-line import/no-extraneous-dependencies | ||
import { Redshift } from 'aws-sdk'; | ||
|
||
const redshift = new Redshift(); | ||
|
||
export async function handler(event: AWSLambda.CloudFormationCustomResourceEvent): Promise<void> { | ||
if (event.RequestType !== 'Delete') { | ||
return rebootClusterIfRequired(event.ResourceProperties?.ClusterId, event.ResourceProperties?.ParameterGroupName); | ||
} else { | ||
return; | ||
} | ||
} | ||
|
||
async function rebootClusterIfRequired(clusterId: string, parameterGroupName: string): Promise<void> { | ||
return executeActionForStatus(await getApplyStatus()); | ||
|
||
// https://docs.aws.amazon.com/redshift/latest/APIReference/API_ClusterParameterStatus.html | ||
async function executeActionForStatus(status: string, retryDurationMs?: number): Promise<void> { | ||
await sleep(retryDurationMs ?? 0); | ||
if (['pending-reboot', 'apply-deferred', 'apply-error'].includes(status)) { | ||
try { | ||
await redshift.rebootCluster({ ClusterIdentifier: clusterId }).promise(); | ||
} catch (err) { | ||
if ((err as any).code === 'InvalidClusterState') { | ||
return await executeActionForStatus(status, 30000); | ||
} else { | ||
throw err; | ||
} | ||
} | ||
return; | ||
} else if (['applying', 'retry'].includes(status)) { | ||
return executeActionForStatus(await getApplyStatus(), 30000); | ||
} | ||
return; | ||
} | ||
|
||
async function getApplyStatus(): Promise<string> { | ||
const clusterDetails = await redshift.describeClusters({ ClusterIdentifier: clusterId }).promise(); | ||
if (clusterDetails.Clusters?.[0].ClusterParameterGroups === undefined) { | ||
throw new Error(`Unable to find any Parameter Groups associated with ClusterId "${clusterId}".`); | ||
} | ||
for (const group of clusterDetails.Clusters?.[0].ClusterParameterGroups) { | ||
if (group.ParameterGroupName === parameterGroupName) { | ||
return group.ParameterApplyStatus ?? 'retry'; | ||
} | ||
} | ||
throw new Error(`Unable to find Parameter Group named "${parameterGroupName}" associated with ClusterId "${clusterId}".`); | ||
} | ||
} | ||
|
||
function sleep(ms: number) { | ||
return new Promise(resolve => setTimeout(resolve, ms)); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
...napshot/asset.1b88b7c3e3e0f8d3e27ded1bde51b7a80c75f3d8733872af7952c3a6d902147e/index.d.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
export declare function handler(event: AWSLambda.CloudFormationCustomResourceEvent): Promise<void>; |
56 changes: 56 additions & 0 deletions
56
....snapshot/asset.1b88b7c3e3e0f8d3e27ded1bde51b7a80c75f3d8733872af7952c3a6d902147e/index.js
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
53 changes: 53 additions & 0 deletions
53
....snapshot/asset.1b88b7c3e3e0f8d3e27ded1bde51b7a80c75f3d8733872af7952c3a6d902147e/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
// eslint-disable-next-line import/no-extraneous-dependencies | ||
import { Redshift } from 'aws-sdk'; | ||
|
||
const redshift = new Redshift(); | ||
|
||
export async function handler(event: AWSLambda.CloudFormationCustomResourceEvent): Promise<void> { | ||
if (event.RequestType !== 'Delete') { | ||
return rebootClusterIfRequired(event.ResourceProperties?.ClusterId, event.ResourceProperties?.ParameterGroupName); | ||
} else { | ||
return; | ||
} | ||
} | ||
|
||
async function rebootClusterIfRequired(clusterId: string, parameterGroupName: string): Promise<void> { | ||
return executeActionForStatus(await getApplyStatus()); | ||
|
||
// https://docs.aws.amazon.com/redshift/latest/APIReference/API_ClusterParameterStatus.html | ||
async function executeActionForStatus(status: string, retryDurationMs?: number): Promise<void> { | ||
await sleep(retryDurationMs ?? 0); | ||
if (['pending-reboot', 'apply-deferred', 'apply-error', 'unknown-error'].includes(status)) { | ||
try { | ||
await redshift.rebootCluster({ ClusterIdentifier: clusterId }).promise(); | ||
} catch (err) { | ||
if ((<any>err).code === 'InvalidClusterState') { | ||
return await executeActionForStatus(status, 30000); | ||
} else { | ||
throw err; | ||
} | ||
} | ||
return; | ||
} else if (['applying', 'retry'].includes(status)) { | ||
return executeActionForStatus(await getApplyStatus(), 30000); | ||
} | ||
return; | ||
} | ||
|
||
async function getApplyStatus(): Promise<string> { | ||
const clusterDetails = await redshift.describeClusters({ ClusterIdentifier: clusterId }).promise(); | ||
if (clusterDetails.Clusters?.[0].ClusterParameterGroups === undefined) { | ||
throw new Error(`Unable to find any Parameter Groups associated with ClusterId "${clusterId}".`); | ||
} | ||
for (const group of clusterDetails.Clusters?.[0].ClusterParameterGroups) { | ||
if (group.ParameterGroupName === parameterGroupName) { | ||
return group.ParameterApplyStatus ?? 'retry'; | ||
} | ||
} | ||
throw new Error(`Unable to find Parameter Group named "${parameterGroupName}" associated with ClusterId "${clusterId}".`); | ||
} | ||
} | ||
|
||
function sleep(ms: number) { | ||
return new Promise(resolve => setTimeout(resolve, ms)); | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought about doing some sort of backoff here, but found checking every
30
seconds to be sufficient in my testing.