bug: File closed twice #17549
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
2024-04-08T10:51:14Z ERR error from StreamingBatchWriter error="handler failed on test_testdata_table: close test_testdata_table.parquet: file already closed" invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=file
Expected Behavior
No issue on the file writing
CloudQuery (redacted) config
kind: source
spec:
name: test
registry: "github"
path: "cloudquery/test"
version: "v4.1.0"
tables:
- "test_testdata_table"
destinations:
- file
---
kind: destination
spec:
name: "file"
path: "cloudquery/file"
registry: "cloudquery"
version: "v4.0.6"
write_mode: "append"
spec:
path: ./{{TABLE}}.{{FORMAT}}
format: "parquet"
no_rotate: trueSteps To Reproduce
No response
CloudQuery (redacted) logs
Logs
2024-04-08T10:51:07Z INF Loading spec(s) args=["./file.yaml","./test.yaml"] invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:07Z WRN registry: github is deprecated & will be removed in future releases invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 kind=source module=cli name=test
2024-04-08T10:51:09Z INF Plugin server listening address=/var/folders/xq/3kqvky950gs9yxllghq1cw2r0000gn/T/cq-qKwpeViRnMWwlcwG.sock invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:13Z INF Plugin server listening address=/var/folders/xq/3kqvky950gs9yxllghq1cw2r0000gn/T/cq-tGyJxGZzENJoiGrB.sock invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.076 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.176 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.036 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.136 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF Start sync destinations=["file (cloudquery/[email protected])"] invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli source="test@github (cloudquery/[email protected])" sync_time=2024-04-08T10:51:14Z
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.01 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.072 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.013 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.627 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.022 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.082 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.109 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=6.309 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF Start fetching resources destinations=["file (cloudquery/[email protected])"] invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli source="test@github (cloudquery/[email protected])"
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.078 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF top level table resolver started client=Client#2 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=test-src table=test_testdata_table
2024-04-08T10:51:14Z INF table sync finished client=Client#2 errors=0 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=test-src resources=1 table=test_testdata_table
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=2.929 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=4.489 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z ERR error from StreamingBatchWriter error="handler failed on test_testdata_table: close test_testdata_table.parquet: file already closed" invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=file
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=18.937 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.005 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.024 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF End sync destinations=["file (cloudquery/[email protected])"] invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli source="test@github (cloudquery/[email protected])" sync_time=2024-04-08T10:51:14Z
2024-04-08T10:51:14Z INF waiting for source plugin to terminate invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:14Z INF Got stop signal. Plugin server shutting down address=/var/folders/xq/3kqvky950gs9yxllghq1cw2r0000gn/T/cq-tGyJxGZzENJoiGrB.sock invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli signal=interrupt
2024-04-08T10:51:14Z INF waiting for source plugin to terminate invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:14Z INF Got stop signal. Plugin server shutting down address=/var/folders/xq/3kqvky950gs9yxllghq1cw2r0000gn/T/cq-qKwpeViRnMWwlcwG.sock invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli signal=interrupt
CloudQuery version
5.11.0
Additional Context
No response
Pull request (optional)
- I can submit a pull request