Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(redshift): optionally reboot Clusters to apply parameter changes #22063

Merged
merged 25 commits into from
Feb 17, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
8b8798c
inital implementation
dontirun Sep 13, 2022
68ebe28
add lambda handler
dontirun Sep 14, 2022
f262a52
documentation, tests, and bugfixes
Sep 15, 2022
6c1122b
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
Sep 30, 2022
c61f994
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
Oct 4, 2022
6fe9387
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
Oct 10, 2022
63400f6
fix: allow for enabling the reboot feature before the parameter group…
Oct 10, 2022
a5d1ee5
test: comment out assertions
Oct 10, 2022
153345a
chore: commit deleted asstes
Oct 10, 2022
f31b6dd
chore: update snapshot
Oct 11, 2022
a01c334
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
Oct 19, 2022
52916e6
tests: regenerate snapshot
Oct 19, 2022
c445b0f
chore: commenting out diff assets in integ test
Nov 16, 2022
26fbbaa
docs: add testing procedure to integ test
Nov 16, 2022
b732dcf
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
Nov 16, 2022
692d1ab
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
Dec 9, 2022
762f2bf
test: update assertions on integ test
Dec 14, 2022
dae7883
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
dontirun Dec 16, 2022
75ebd28
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
dontirun Jan 18, 2023
b6c323d
refactor: removing unknown-error from reboot actions
dontirun Feb 14, 2023
1c69185
Merge branch 'master' into feat-reboot-cluster-for-parameter-updates
dontirun Feb 14, 2023
80848a9
Update packages/@aws-cdk/aws-redshift/lib/cluster-parameter-change-re…
dontirun Feb 14, 2023
f91ffeb
refactor: use guard clause for clarity
Feb 16, 2023
a76b887
refactor: update guard clause
Feb 17, 2023
16ba85f
Merge branch 'main' into feat-reboot-cluster-for-parameter-updates
mergify[bot] Feb 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
inital implementation
  • Loading branch information
dontirun committed Sep 13, 2022
commit 8b8798c2ae69ed098d92cd8aabf020d5f235eed4
70 changes: 68 additions & 2 deletions packages/@aws-cdk/aws-redshift/lib/cluster.ts
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
import * as path from 'path';
import * as ec2 from '@aws-cdk/aws-ec2';
import * as iam from '@aws-cdk/aws-iam';
import * as kms from '@aws-cdk/aws-kms';
import * as lambda from '@aws-cdk/aws-lambda';
import * as s3 from '@aws-cdk/aws-s3';
import * as secretsmanager from '@aws-cdk/aws-secretsmanager';
import { Duration, IResource, RemovalPolicy, Resource, SecretValue, Token } from '@aws-cdk/core';
import { Duration, IResource, RemovalPolicy, Resource, SecretValue, Token, CustomResource, Stack, ArnFormat } from '@aws-cdk/core';
import * as cr from '@aws-cdk/custom-resources';
import { Construct } from 'constructs';
import { DatabaseSecret } from './database-secret';
import { Endpoint } from './endpoint';
import { ClusterParameterGroup, IClusterParameterGroup } from './parameter-group';
import { CfnCluster } from './redshift.generated';
import { ClusterSubnetGroup, IClusterSubnetGroup } from './subnet-group';

/**
* Possible Node Types to use in the cluster
* used for defining {@link ClusterProps.nodeType}.
Expand Down Expand Up @@ -354,6 +356,12 @@ export interface ClusterProps {
* @default - No Elastic IP
*/
readonly elasticIp?: string

/**
* If this flag is set, the cluster will be rebooted when changes to the cluster's parameter group that require a restart to apply.
* @default false
*/
readonly rebootForParameterChanges?: boolean
}

/**
Expand Down Expand Up @@ -452,6 +460,11 @@ export class Cluster extends ClusterBase {
*/
protected parameterGroup?: IClusterParameterGroup;

/**
* Whether the cluster will be rebooted when changes to the cluster's parameter group that require a restart to apply.
*/
protected rebootForParameterChangesEnabled?: boolean;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this prop, can we remove it? It seems redundant with rebootForParameterChanges?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to my other comment rebootForParameterChanges is the class property that is needed so we don't create duplicate custom resources with enableRebootForParameterChanges(). I thought it would be clearer to have this property versus making the Provider and CustomResource class properties

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add to this docstring that this is only used to guard against repeated invocations of enableRebootForParameterChanges()


constructor(scope: Construct, id: string, props: ClusterProps) {
super(scope, id);

Expand Down Expand Up @@ -565,6 +578,9 @@ export class Cluster extends ClusterBase {

const defaultPort = ec2.Port.tcp(this.clusterEndpoint.port);
this.connections = new ec2.Connections({ securityGroups, defaultPort });
if (props.rebootForParameterChanges) {
this.enableRebootForParameterChanges();
}
}

/**
Expand Down Expand Up @@ -652,4 +668,54 @@ export class Cluster extends ClusterBase {
throw new Error('Cannot add a parameter to an imported parameter group.');
}
}

/**
* Enables automatic cluster rebooting when changes to the cluster's parameter group require a restart to apply.
*/
public enableRebootForParameterChanges(): void {
if (!this.parameterGroup) {
throw new Error('Cannot enable reboot for parameter changes when there is no associated ClusterParameterGroup.');
}
if (!(this.parameterGroup instanceof ClusterParameterGroup)) {
throw new Error('Cannot enable reboot for parameter changes when using an imported parameter group.');
}
if (!this.rebootForParameterChangesEnabled) {
this.rebootForParameterChangesEnabled = true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check on line 607 is all we need for this, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we need it to make sure we don't make a duplicate Custom Resource. I could create class variables to make the function idempotent, but thought that may be overkill

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

making the function idempotent is definitely overkill, and guarding against these potential bugs (an error would be thrown complaining about duplicate construct IDs) is definitely the right call in general. In this case though I don't see how the function could be called twice, as it's only called in the constructor and it's internal-only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally thought about making it constructor only, but It's currently not (example). I figured that exposing the method would be friendlier for situationally enabling the feature under specific conditions (ex. beta/prod stages, using the addToParameterGroup method and then using this method).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm okay. I don't love that we have this prop and this internal member, but it is a better user experience to be able to use a convenience method to set it, so we can leave it.

Please change the format of this to a guard clause though, eg:

    if (this.rebootForParameterChangesEnabled) {
      return;
    }
    this.rebootForParameterChangesEnabled = true;

const rebootFunction = new lambda.SingletonFunction(this, 'RedshiftClusterRebooterFunction', {
uuid: '511e207f-13df-4b8b-b632-c32b30b65ac2',
runtime: lambda.Runtime.NODEJS_16_X,
code: lambda.Code.fromAsset(path.join(__dirname, 'cluster-parameter-change-reboot-handler')),
handler: 'index.handler',
timeout: Duration.seconds(900),
});
rebootFunction.addToRolePolicy(new iam.PolicyStatement({
actions: ['redshift:DescribeClusters'],
resources: ['*'],
}));
rebootFunction.addToRolePolicy(new iam.PolicyStatement({
actions: ['reshift:RebootCluster'],
resources: [
Stack.of(this).formatArn({
service: 'redshift',
resource: 'cluster',
resourceName: this.clusterName,
arnFormat: ArnFormat.COLON_RESOURCE_NAME,
}),
],
}));
const provider = new cr.Provider(this, 'ResourceProvider', {
onEventHandler: rebootFunction,
});
const customResource = new CustomResource(this, 'RedshiftClusterRebooterCustomResource', {
resourceType: 'Custom::RedshiftClusterRebooter',
serviceToken: provider.serviceToken,
properties: {
ClusterId: this.cluster.getAtt('id'),
ParameterGroupName: this.parameterGroup.clusterParameterGroupName,
ParametersString: JSON.stringify(this.parameterGroup.parameters),
},
});
customResource.node.addDependency(this, this.parameterGroup);
}
}
}
74 changes: 74 additions & 0 deletions packages/@aws-cdk/aws-redshift/test/cluster.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -613,6 +613,80 @@ test('elastic ip address', () => {
});
});

describe('reboot for Parameter Changes', () => {
test('cluster without parameter group', () => {
// Given
const cluster = new Cluster(stack, 'Redshift', {
masterUser: {
masterUsername: 'admin',
},
vpc,
});

// WHEN
expect(() => cluster.enableRebootForParameterChanges())
// THEN
.toThrowError(/Cannot enable reboot for parameter changes/);
});

test('cluster with imported parameter group', () => {
// Given
const cluster = new Cluster(stack, 'Redshift', {
masterUser: {
masterUsername: 'admin',
},
vpc,
});
cluster.addToParameterGroup('foo', 'bar');

const cluster2 = new Cluster(stack, 'Redshift2', {
masterUser: {
masterUsername: 'admin',
},
vpc,
});
cluster2.addToParameterGroup('foo', 'bar');

// WHEN
cluster.enableRebootForParameterChanges();
cluster2.enableRebootForParameterChanges();

//THEN
const template = Template.fromStack(stack);
template.resourceCountIs('Custom::RedshiftClusterRebooter', 2);
template.templateMatches({
Resources: {
SingletonLambda511e207f13df4b8bb632c32b30b65ac281740AC5: {
Type: 'AWS::Lambda::Function',
Properties: {
Handler: 'index.handler',
Runtime: 'nodejs16.x',
Timeout: 900,
},
},
},
});
});


test('cluster with parameter group', () => {
// Given
const cluster = new Cluster(stack, 'Redshift', {
masterUser: {
masterUsername: 'admin',
},
vpc,
parameterGroup: ClusterParameterGroup.fromClusterParameterGroupName(stack, 'foo', 'bar'),
});

// WHEN
expect(() => cluster.enableRebootForParameterChanges())
// THEN
.toThrowError(/Cannot enable reboot for parameter changes/);
});

});

function testStack() {
const newTestStack = new cdk.Stack(undefined, undefined, { env: { account: '12345', region: 'us-test-1' } });
newTestStack.node.setContext('availability-zones:12345:us-test-1', ['us-test-1a', 'us-test-1b']);
Expand Down