Skip to content

Incremental macros missing from packaged dbt-spark==0.19.0 #152

Closed
@highfly22

Description

@highfly22

Describe the bug

When a incremental model is running, the following error is reported

  Runtime Error
    Database Error
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'my_table__dbt_tmp' not found in database 'FooSchema';

For example, the schema is configured as "FooSchema". The temporary view is created as "my_table__dbt_tmp". The describe command tries to find 'FooSchema.my_table__dbt_tmp', which doesn't exists.

    create temporary view my_table__dbt_tmp as
    
    describe extended FooSchema.my_table__dbt_tmp

Steps To Reproduce

  1. configure spark profile
  2. create a incremental model
  3. invoke dbt run -m my_table twice

Expected behavior

No error is reported.

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

The output of dbt --version:

installed version: 0.19.0
   latest version: 0.19.0

Up to date!

Plugins:
  - spark: 0.19.0
  - postgres: 0.19.0
  - redshift: 0.19.0
  - snowflake: 0.19.0
  - bigquery: 0.19.0

Python 3.6.3

The operating system you're using:

The output of python --version:
Python 3.6.3

Additional context

    dev:
      type: spark
      method: thrift
      schema: FooSchema
      host: localhost
      port: 10000
      user: Foo
      auth: NONE

Thrift server is created by spark 2.4.7

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions