Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(msk): add Kafka versions 3.1.1, 3.2.0, and and 3.3.1 #23918

Merged
merged 14 commits into from
Feb 22, 2023

Conversation

stefanfreitag
Copy link
Contributor

Add support for Apache Kafka versions 3.1.1, 3.2.0 and 3.3.1 in Amazon MSK.

Announcements:

Closes #23899

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

stefanfreitagrwe and others added 2 commits January 30, 2023 21:38
Added support for Kafka versions 3.1.1, 3.2.0, and 3.3.1
@gitpod-io
Copy link

gitpod-io bot commented Jan 30, 2023

@github-actions github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK feature-request A feature should be added or improved. p2 labels Jan 30, 2023
@aws-cdk-automation aws-cdk-automation requested a review from a team January 30, 2023 20:52
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

@stefanfreitag
Copy link
Contributor Author

The code change adds new versions to the already existing/ supported Kafka versions.

  • The supported versions were not mentioned in the README so far - I decided to not start with this.
  • The unit/ integration tests all use the default version 2.6.1 of Kafka. I thought about updating those tests, but skipped. Happy to apply a change (new PR or as part of this).
    Nevertheless, if we go with the default/ recommended version it would be in my opinion 2.8.1 - the newly introduced versions would also not be covered by this.

Please let me know your thoughts.

@TheRealAmazonKendra TheRealAmazonKendra added the pr-linter/exempt-readme The PR linter will not require README changes label Feb 1, 2023
@TheRealAmazonKendra
Copy link
Contributor

The code change adds new versions to the already existing/ supported Kafka versions.

  • The supported versions were not mentioned in the README so far - I decided to not start with this.
  • The unit/ integration tests all use the default version 2.6.1 of Kafka. I thought about updating those tests, but skipped. Happy to apply a change (new PR or as part of this).
    Nevertheless, if we go with the default/ recommended version it would be in my opinion 2.8.1 - the newly introduced versions would also not be covered by this.

Please let me know your thoughts.

Thank you for your contribution!

I've made a README exemption. I don't think that it's necessary in this case.

It's unfortunate that we have a testing gap here that we didn't notice, but I'd like to not increase that gap. I'd like to see test cases added for the new versions (not replacing the old ones). If running integ tests is a blocker for any reason, please let us know and we can run them once the test cases themselves are written.

@stefanfreitag
Copy link
Contributor Author

Hi @TheRealAmazonKendra !
I tried to come up with a unit test for the Kafka versions. To my understanding we would need to check/ensure that msk.KafkaVersion.VX_Y_Z contains always the correct version as string, e.g. msk.KafkaVersion.V3_2_0 -> '3.2.0'.
Not sure if a parameterized test is what you had in mind, but its a starting point for discussion.
(ok, noticed that there is a different flavor for the syntax around test.each - will have a look)

On the integration test I am unsure about the expectation. I noticed that three clusters with Kafka version 2.8.1 are generated, all with a different way for e.g. authentication. Would i need to setup an additional cluster with the newer Kafka versions? Or three because of the different authentication mechanisms?

@aws-cdk-automation aws-cdk-automation dismissed their stale review February 12, 2023 07:59

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

@stefanfreitag
Copy link
Contributor Author

@TheRealAmazonKendra I was able to run the integration tests in an AWS account and notices that it creates a Private CA - which is quite costly. Is there any information how contributors typically deal with this kind of costs?

@TheRealAmazonKendra
Copy link
Contributor

Hi @TheRealAmazonKendra ! I tried to come up with a unit test for the Kafka versions. To my understanding we would need to check/ensure that msk.KafkaVersion.VX_Y_Z contains always the correct version as string, e.g. msk.KafkaVersion.V3_2_0 -> '3.2.0'. Not sure if a parameterized test is what you had in mind, but its a starting point for discussion. (ok, noticed that there is a different flavor for the syntax around test.each - will have a look)

On the integration test I am unsure about the expectation. I noticed that three clusters with Kafka version 2.8.1 are generated, all with a different way for e.g. authentication. Would i need to setup an additional cluster with the newer Kafka versions? Or three because of the different authentication mechanisms?

Apologies for the delay, I missed the notification last week. Just one for each. We just want to see that it deploys.

@TheRealAmazonKendra
Copy link
Contributor

@TheRealAmazonKendra I was able to run the integration tests in an AWS account and notices that it creates a Private CA - which is quite costly. Is there any information how contributors typically deal with this kind of costs?

Yes, and we need to make the documentation better because this question comes up frequently so I apologize for it not being more clear. Just write the test cases and commit them. Then tag me with a note to run the tests and update the snapshots for you.

Copy link
Contributor

@TheRealAmazonKendra TheRealAmazonKendra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting back into changes requested to reflect status.

@mergify mergify bot dismissed TheRealAmazonKendra’s stale review February 17, 2023 19:52

Pull request has been modified.

@stefanfreitag
Copy link
Contributor Author

Hello @TheRealAmazonKendra! As discussed I updated the integration test code and added three MSK clusters, one for each version added. Please let me know if I need to change/ update.

@TheRealAmazonKendra
Copy link
Contributor

@Mergifyio update

@mergify
Copy link
Contributor

mergify bot commented Feb 21, 2023

update

❌ Base branch update has failed

refusing to allow a GitHub App to create or update workflow .github/workflows/yarn-upgrade-v1main.yml without workflows permission
err-code: EDDB9

@TheRealAmazonKendra
Copy link
Contributor

Well, this exercise was worthwhile because I found that our test was actually broken. I've pushed the fix and the updated snapshots. As long as this passes, I see it as good to go.

@mergify
Copy link
Contributor

mergify bot commented Feb 22, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 0b293fe
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 53a1d5f into aws:main Feb 22, 2023
@mergify
Copy link
Contributor

mergify bot commented Feb 22, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Naumel pushed a commit that referenced this pull request Feb 22, 2023
Add support for Apache Kafka versions 3.1.1, 3.2.0 and 3.3.1 in Amazon MSK.

Announcements:
- [3.1.1 and 3.2.0](https://aws.amazon.com/about-aws/whats-new/2022/06/amazon-msk-adds-support-apache-kafka-version-3-1-1-3-2-0/) (Posted On: Jun 22, 2022)
- [3.3.1](https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-msk-support-apache-kafka-version-3-3-1) (Posted On: Oct 26, 2022)

Closes #23899 
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Naumel pushed a commit that referenced this pull request Feb 24, 2023
Add support for Apache Kafka versions 3.1.1, 3.2.0 and 3.3.1 in Amazon MSK.

Announcements:
- [3.1.1 and 3.2.0](https://aws.amazon.com/about-aws/whats-new/2022/06/amazon-msk-adds-support-apache-kafka-version-3-1-1-3-2-0/) (Posted On: Jun 22, 2022)
- [3.3.1](https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-msk-support-apache-kafka-version-3-3-1) (Posted On: Oct 26, 2022)

Closes #23899 
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
beck3905 pushed a commit to beck3905/aws-cdk that referenced this pull request Feb 28, 2023
Add support for Apache Kafka versions 3.1.1, 3.2.0 and 3.3.1 in Amazon MSK.

Announcements:
- [3.1.1 and 3.2.0](https://aws.amazon.com/about-aws/whats-new/2022/06/amazon-msk-adds-support-apache-kafka-version-3-1-1-3-2-0/) (Posted On: Jun 22, 2022)
- [3.3.1](https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-msk-support-apache-kafka-version-3-3-1) (Posted On: Oct 26, 2022)

Closes aws#23899 
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
homakk pushed a commit to homakk/aws-cdk that referenced this pull request Mar 13, 2023
Add support for Apache Kafka versions 3.1.1, 3.2.0 and 3.3.1 in Amazon MSK.

Announcements:
- [3.1.1 and 3.2.0](https://aws.amazon.com/about-aws/whats-new/2022/06/amazon-msk-adds-support-apache-kafka-version-3-1-1-3-2-0/) (Posted On: Jun 22, 2022)
- [3.3.1](https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-msk-support-apache-kafka-version-3-3-1) (Posted On: Oct 26, 2022)

Closes aws#23899 
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@stefanfreitag stefanfreitag deleted the 23899 branch March 24, 2023 16:30
homakk pushed a commit to homakk/aws-cdk that referenced this pull request Mar 28, 2023
Add support for Apache Kafka versions 3.1.1, 3.2.0 and 3.3.1 in Amazon MSK.

Announcements:
- [3.1.1 and 3.2.0](https://aws.amazon.com/about-aws/whats-new/2022/06/amazon-msk-adds-support-apache-kafka-version-3-1-1-3-2-0/) (Posted On: Jun 22, 2022)
- [3.3.1](https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-msk-support-apache-kafka-version-3-3-1) (Posted On: Oct 26, 2022)

Closes aws#23899 
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK feature-request A feature should be added or improved. p2 pr-linter/exempt-readme The PR linter will not require README changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(aws-msk): Support newer versions of Kafka
4 participants