-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kafka ingestion source ingests schemas for all topics and ignores topic_patterns #11907
Comments
@hsheth2, if you want i could prepare a PR. From what i see this could be solved by introducing a new config schema_pattern similar to the topic_pattern. Just reusing topic_pattern might be problematic for some topic to schema mapping. Ingesting all schemas separately to the topics where they are already available also doesn't provide us any benefit i think and this would allow to virtually disable the separate ingestion. What do you think? |
@guentherhackl-wgs that sounds like it would be super helpful - Also note that we're planning on adding a flag to disable the ingestion of all schemas separately - we'll probably make a change so that feature is not enabled by default |
@hsheth2 this flag would also do what i'm looking for. Is this already planned for a specific timeframe, as I could also incorporate it? With my suggestion I was mainly thinking of replicating what already exists in topic_pattern and "deny: '*'" which would also disable the feature completely and of course could be set as the default as well, but nothing would speak against a separate flag to disable it |
@guentherhackl-wgs we are already working on the flag, it should be merged shortly |
@guentherhackl-wgs here is the PR, would you be able to try it before we merge it? |
Closing for now since #12077 was merged |
Describe the bug
When i load kafka topics and use topic_patterns it filters all the topics but correctly but still produces events for schemas for all topics on the cluster.
To Reproduce
Expected behavior
the schemas schould also be filtered
Desktop (please complete the following information):
The text was updated successfully, but these errors were encountered: