-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(Kubernetes) Add metadata-ingestion as a Helm component #2236
feat(Kubernetes) Add metadata-ingestion as a Helm component #2236
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall, looks great. Has all the configurations we need.
Minor comment: related to the comment below. Can we name this component to indicate that this is a cronjob based ingestion?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Thank you @dexter-mh-lee! |
I will try it out right now! |
@pedro93 Works!! Thanks for working on this! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks for this contribution @pedro93
Checklist
Adds metadata ingestion as a disabled-by-default helm component of Datahub's kubernetes deployment.
This component will render a Kubernetes CronJob resource for each cron resource specified.
Metadata-ingestion configuration recipes must be supplied by the consumer of this chart as a mounted volume and then explicitly referenced by the
crawlerConfigPath
.An example is as follows:
With the appropriate ConfigMaps holding each concrete recipes deployed to Kubernetes, this example will generate 2 separate cronJob resources in Kubernetes, allowing for reuse of this component for however may metadata ingestion cron jobs may be required.
Test method: