Skip to content

DBT initialization error in CLI ingestion #11931

@dcalcadawork

Description

@dcalcadawork

Describe the bug
When using the below recipe template (in which the metadata paths are working as expected and sink connection is established), I receive an error message as PATH/Python312\site-packages\datahub\ingestion\run\pipeline.py", line 140, in _add_init_error_context
raise PipelineInitError(f"Failed to {step}: {e}") from e
datahub.ingestion.run.pipeline.PipelineInitError: Failed to find a registered source for type dbt: dbt is disabled due to an error in initialization

pipeline_name: "[PIPELINE_NAME]"
source:
type: "dbt"
config:
# Coordinates
manifest_path: "[PATH_TO_MANIFEST.JSON]" #MANIFEST metadata
catalog_path: "[PATH_TO_CATALOG.JSON]" #CATALOG metadata
#sources_path: "[PATH_TO_SOURCES.JSON]" #SOURCES metadata
test_results_path: "[PATH_TO_RESULTS.JSON]" #RESULTS metadata
#only_include_if_in_catalog: true

# Options
target_platform: "bigquery" # e.g. bigquery/postgres/etc.
stateful_ingestion: 
  enabled: true
  remove_stale_metadata: false
node_name_pattern:  # allows for inclusion or exclusion of assets for the ingestion via regular expressions
  ignoreCase: true
  allow:
    - .*[]
  deny:
    - .*[]

#Setup connection to datahub env
sink:
type: "datahub-rest"
config:
server: "[LOCALHOST]"
token: "[ACCESS TOKEN]"

To Reproduce
Steps to reproduce the behavior:

  1. Using virtual environment with wheel
  2. Using DataHub CLI version: 0.14.1.6
  3. 'acryl-datahub[dbt]' is installed
  4. Run ingestion as: datahub ingest -c [PATH]\dbt_meta_ingest.yaml

Expected behavior
Ingestion Pipeline finished

Screenshots
image

Desktop (please complete the following information):

  • OS: WINDOWS 10
  • Command line: Windows powershell

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportingestionPR or Issue related to the ingestion of metadata

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions