Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Hive metadata ETL process #9

Merged
merged 2 commits into from
Dec 21, 2015
Merged

Conversation

SunZhaonan
Copy link
Contributor

ETL of Hive metadata into Wherehows repository.
Two major sources from hive meta-store tables: COLUMN_V2 , SERDE_PARAM

Things haven't done yet (left for later) :

  1. Field didn't load into dict_field_detail table. But have schema json info in the dict_dataset.
  2. View dependency didn't handle the nested View dependency yet.
  3. Sample data.

@@ -67,5 +67,7 @@ subprojects {
"play" : "com.typesafe.play:play_2.10:2.2.4",
"play_ebean" : "com.typesafe.play:play-java-ebean_2.10:2.2.4",
"play_java_jdbc" : "com.typesafe.play:play-java-jdbc_2.10:2.2.4",
"play_cache" : "com.typesafe.play:play-cache_2.10:2.2.4"]
"play_cache" : "com.typesafe.play:play-cache_2.10:2.2.4",
"hive_exec" : "org.apache.hive:hive-exec:0.8.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this version of hive-exec chosen for specific reason?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No specific reason, I just choose a random one as their interface doesn't have too much changes. Maybe it's better to change to a latest one.

czbernard added a commit that referenced this pull request Dec 21, 2015
Add Hive metadata ETL process
@czbernard czbernard merged commit 348385f into datahub-project:master Dec 21, 2015
Masterchen09 pushed a commit to Masterchen09/datahub that referenced this pull request Jul 14, 2022
Masterchen09 pushed a commit to Masterchen09/datahub that referenced this pull request Sep 4, 2022
aabharti-visa pushed a commit to aabharti-visa/datahub that referenced this pull request Feb 21, 2024
…butes-propagation

business attribute: businessattributes-propagation
jeff-xu-z pushed a commit to jeff-xu-z/datahub that referenced this pull request Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants