feat(ingest): add support for Looker view built from SQL-based derived tables #2478
Conversation
Parse the SQL query in the derived_table field of Looker views to extract upstream tables. Dicard temporary tables created with the SQL WITH statement. Update the lkml dependency to parse .lkml files without throwing errors when field values contain special characters.
There was a problem hiding this comment.
LGTM - thanks @remisalmon!
@frsann is this OK to merge before you port the Looker integration into the main ingestion framework?
|
Yes, go ahead! I haven't started yet! Awesome stuff here! |
|
A quick question, from a Looker newbie: are the sql queries in the derived tables expressed in the dialect of the "backend" database? I was curious about this |
|
@frsann the answer to your question seems to be yes: https://docs.looker.com/reference/view-params/derived_table#supported_database_dialects_for_derived_tables I also found an issue with |
|
Considering that the sql parsing might not be water tight, should we put this behind feature flag? |
Agree here -- seems like the sql-metadata libs only supports a few dialects, whereas Looker appears to support many more? Have we considered whether something like sqlparse is suitable? |
|
I'm going to merge this in first. We can improve sql parsing in a follow up PR. Want to first prioritize the moving of this to the ingestion framework. |
|
@jjoyce0510 sql-metadata actually uses sqlparse under the hood. Agree that we should have put this behind a feature flag since it's not yet fully baked. |
|
@hsheth2 @jjoyce0510 I am trying to get some |
This PR adds support for ingesting Looker views built from SQL-based derived tables (https://docs.looker.com/data-modeling/learning-lookml/derived-tables).
Tested with success on our Looker instance, with the lineage of Looker views built from SQL-based derived tables showing up in DataHub.
Also updates the lkml dependency to better parse .lkml files.
Checklist