Skip to content

Commit

Permalink
feat(ingest): adds experimental support for ingesting Looker metadata (
Browse files Browse the repository at this point in the history
  • Loading branch information
grantatspothero authored Apr 2, 2021
1 parent 9d00e8b commit fce8a53
Show file tree
Hide file tree
Showing 6 changed files with 801 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
## looker_dashboard_ingestion.py
This tool helps ingest Looker dashboard and chart metadata into datahub.
Currently it creates a separate platform named "looker" and loads all dashboard and chart information into that platform as virtual datasets. This was to workaround datahub's lack of support for dashboard entities, however datahub recently started supporting proper dashboard entities.

The script assumes you already have run lookml_ingestion.py to scrape view definitions into datahub, this is important because we assign lineage between looker views and looker dashboards/charts where possible.


## Steps:
- Use a version of python >= 3.7
- Make a virtual environment
- pip install -r requirements.txt
- Set env vars: LOOKERSDK_CLIENT_ID, LOOKERSDK_CLIENT_SECRET, LOOKERSDK_BASE_URL
- Configure extra kafka conf in looker_dashboard_ingestion.py

python looker_dashboard_ingestion.py
Loading

0 comments on commit fce8a53

Please sign in to comment.