Skip to content

bug: window functions not working as expected in BigQuery backend #10390

Closed
@efcaguab

Description

@efcaguab

What happened?

When I try to use nunique with a group_by, or window I'm getting an error suggesting that the SQL is not allowed by BigQuery.

The same ibis expressions seem to work well using duckdb. Perhaps something to do with the row-wise arrangement of the window function?

import ibis
from ibis import _
import pandas as pd

bq_dataset = "scratch_fer"
data = pd.DataFrame({
    "foo": [1, 1, 2, 2, 3, 3],
    "bar": ["a", "b", "a", "a", "b", "b"]
})
bq = ibis.bigquery.connect()
bq.create_table("test", data, database=bq_dataset, overwrite=True)
test_table = bq.table(f"{bq_dataset}.test")

# All these commands below fail: (400 Window framing clause is not allowed if
# DISTINCT is specified at [3:60]; reason: invalidQuery, location: query,
# message: Window framing clause is not allowed if DISTINCT is specified
test_table.group_by("foo").mutate(bar=_.bar.nunique()).to_pandas()
test_table.filter(_.bar.nunique().over(ibis.window(group_by="foo")) > 1).to_pandas()

# However these (which I think will be the equivalent of above) work
test_table.sql(f"SELECT foo, COUNT(DISTINCT bar) OVER (PARTITION BY foo) AS bar FROM {bq_dataset}.test").to_pandas()
test_table.sql(f"SELECT * FROM {bq_dataset}.test QUALIFY COUNT(DISTINCT bar) OVER (PARTITION BY foo) > 1").to_pandas()

What version of ibis are you using?

9.5.0

What backend(s) are you using, if any?

bigquery

Relevant log output

(400 Window framing clause is not allowed if
# DISTINCT is specified at [3:60]; reason: invalidQuery, location: query,
# message: Window framing clause is not allowed if DISTINCT is specified

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIncorrect behavior inside of ibis

    Type

    No type

    Projects

    Status

    done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions