bug: Postgres source is not recognizing partitioned tables #17950
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Loading spec(s) from postgres-to-motherduck.yml
Starting sync for: postgresql (cloudquery/[email protected]) -> [motherduck (cloudquery/[email protected])]
| Syncing resources... (0/-, 0 resources/hr) [0s] Error: failed to sync v3 source postgresql: unexpected error from sync client receive: rpc error: code = Unknown desc = failed to sync records: failed to sync unmanaged client: tables include a pattern mypartitioned_tablename with no matchesExpected Behavior
Loading spec(s) from postgres-to-motherduck.yml
Starting sync for: postgresql (cloudquery/[email protected]) -> [motherduck (cloudquery/[email protected])]
Sync completed successfully. Resources: 1359, Errors: 0, Warnings: 0, Time: 14s
CloudQuery (redacted) config
kind: source
spec:
name: "postgresql"
path: "cloudquery/postgresql"
registry: "cloudquery"
version: "v6.0.2"
destinations: ["motherduck"]
tables: ["mypartitioned_tablename"]
spec:
connection_string: {CONN}
---
kind: destination
spec:
name: "motherduck"
version: "v5.6.2"
registry: "cloudquery"
path: "cloudquery/duckdb"
write_mode: "overwrite-delete-stale"
migrate_mode: "safe"
spec:
connection_string: "md:"
batch_size: 10000
debug: trueSteps To Reproduce
No response
CloudQuery (redacted) logs
2024-05-12T11:35:19Z INF Loading spec(s) args=["postgres-to-motherduck.yml"] invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli
2024-05-12T11:35:20Z INF Plugin server listening address=/var/folders/k_/9rmq3d012v50trs8xlb5xwm80000gn/T/cq-duWvRLmRVUlqsEcf.sock invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli
2024-05-12T11:35:22Z INF Plugin server listening address=/var/folders/k_/9rmq3d012v50trs8xlb5xwm80000gn/T/cq-HYVztPgeSuBFvNDW.sock invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli
2024-05-12T11:35:23Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-05-12T17:05:23+05:30 grpc.time_ms=0.119 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:23Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-05-12T17:05:23+05:30 grpc.time_ms=0.28 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:23Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-05-12T17:05:23+05:30 grpc.time_ms=0.015 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:23Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-05-12T17:05:23+05:30 grpc.time_ms=0.19 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:23Z INF Start sync destinations=["motherduck (cloudquery/[email protected])"] invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli source="postgresql (cloudquery/[email protected])" sync_time=2024-05-12T11:35:23Z
2024-05-12T11:35:23Z INF started call grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:23+05:30 grpc.time_ms=0.003 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:23Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:23+05:30 grpc.time_ms=0.168 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:23Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:23+05:30 grpc.time_ms=0.004 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:26Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:23+05:30 grpc.time_ms=3516.865 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:26Z INF started call grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:26+05:30 grpc.time_ms=0.021 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:26Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:26+05:30 grpc.time_ms=0.273 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:26Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:26+05:30 grpc.time_ms=0.01 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:26Z INF Initializing postgresql destination invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe invocation_id=78cd3570-d942-4021-939d-69fbe0a094fe module=pg-source pgx_log_level="invalid level 0"
2024-05-12T11:35:27Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:26+05:30 grpc.time_ms=768.972 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:27Z INF Start fetching resources destinations=["motherduck (cloudquery/[email protected])"] invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli source="postgresql (cloudquery/[email protected])"
2024-05-12T11:35:27Z INF started call grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:27+05:30 grpc.time_ms=0.035 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:28Z ERR finished call grpc.code=Unknown grpc.component=server grpc.error="failed to sync records: failed to sync unmanaged client: tables include a pattern mypartitioned_tablename with no matches" grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:27+05:30 grpc.time_ms=547.462 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:28Z INF End sync destinations=["motherduck (cloudquery/[email protected])"] invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli source="postgresql (cloudquery/[email protected])" sync_time=2024-05-12T11:35:23Z
2024-05-12T11:35:28Z INF Sending sync summary to analyticsv1.cloudquery.io:443 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli
2024-05-12T11:35:28Z INF waiting for source plugin to terminate invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli
2024-05-12T11:35:28Z INF started call grpc.component=server grpc.error="rpc error: code = Canceled desc = context canceled" grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:27+05:30 grpc.time_ms=858.567 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:28Z ERR finished call grpc.code=Internal grpc.component=server grpc.error="rpc error: code = Internal desc = failed to receive msg: rpc error: code = Canceled desc = context canceled" grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-05-12T17:05:27+05:30 grpc.time_ms=859.462 invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli peer.address= protocol=grpc
2024-05-12T11:35:28Z INF Got stop signal. Plugin server shutting down address=/var/folders/k_/9rmq3d012v50trs8xlb5xwm80000gn/T/cq-HYVztPgeSuBFvNDW.sock invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli signal=interrupt
2024-05-12T11:35:28Z INF waiting for source plugin to terminate invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli
2024-05-12T11:35:28Z INF Got stop signal. Plugin server shutting down address=/var/folders/k_/9rmq3d012v50trs8xlb5xwm80000gn/T/cq-duWvRLmRVUlqsEcf.sock invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli signal=interrupt
Error: failed to sync v3 source postgresql: unexpected error from sync client receive: rpc error: code = Unknown desc = failed to sync records: failed to sync unmanaged client: tables include a pattern mypartitioned_tablename with no matches
2024-05-12T11:35:28Z ERR exiting with error error="failed to sync v3 source postgresql: unexpected error from sync client receive: rpc error: code = Unknown desc = failed to sync records: failed to sync unmanaged client: tables include a pattern mypartitioned_tablename with no matches" invocation-id=78cd3570-d942-4021-939d-69fbe0a094fe module=cli
CloudQuery version
cloudquery version 5.18.1
Additional Context
No response
Pull request (optional)
- I can submit a pull request