Bump up kafkaAvroSerde to support SSL for Schema Registry #1898
Conversation
docs/faq.md
Outdated
| @@ -116,3 +116,6 @@ You can call the [rest.li](https://github.com/linkedin/rest.li) API to ingest me | |||
| ## Does Kafka support SSL? If so, how? | |||
|
|
|||
| Yes. We are using the Spring Boot framework to start our apps, including setting up Kafka. You can [use environment variables to set system properties](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-relaxed-binding-from-environment-variables), including [Kafka properties](https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#integration-properties). From there you can set your SSL configuration for Kafka. | |||
|
|
|||
| If Schema Registry is configured to use security (SSL), then you also need to set the following config: https://docs.confluent.io/current/kafka/encryption.html#encryption-ssl-schema-registry. | |||
| Note! The logs might say something similar to 'kafkastore.ssl.truststore.password was supplied but isn't a known config', but this is wrong and can be ignored. | |||
There was a problem hiding this comment.
Is there a link to a ticket or Slack thread for this known issue?
There was a problem hiding this comment.
I've opened Request #41858 for Confluent asking this, no answer yet.... and yes, needs a clarification before it can be said to be a bug.
There was a problem hiding this comment.
Ok, not wrong log actually. This log
kindred-datahub-datahub-gms-79b6587bb-t62vj datahub-gms 08:49:56.302 [main] WARN o.a.k.c.producer.ProducerConfig - The configuration 'kafkastore.ssl.truststore.password' was supplied but isn't a known config.
means the configuration kafkastore.ssl.truststore.password' is not a configuration required for the producer. These WARN message can be safely ignored. Each of Datahub services are passed a full set of configuration but may not require all the configurations that are passed to them. These warn messages indicate that the service was passed a configuration that is not relevant to it and can be safely ignored.
Btw, did the build pass? Was thinking to update that comment according to explanation above...
There was a problem hiding this comment.
Oops I didn't wait for the CI to pass. Will test and see if our newly added notification (#1873) works :)
There was a problem hiding this comment.
Hi @mars-lan, yes, I saw something failed and wanted to double-check but then saw again that it was merged....
Now I am unsure of the status...
There was a problem hiding this comment.
No problem. Testing the notification and fixing the build at the same time :)
|
closing #1861 |
fix: Support Datahub to connect to Kafka Schema Registry using SSL
Related to: #1861
Test result:
You should also configure Schema Registry to use security (contrib/kubernetes/datahub/values.yaml):
Note! The logs might say something similar to 'kafkastore.ssl.truststore.password was supplied but isn't a known config', but this is wrong and can be ignored.
Added note to docs/FAQ