Skip to content

[fix](parquet)Parse Parquet INT96 physical type data into a no-timezone datetime. #59514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
hubgeter wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[fix](parquet)Parse Parquet INT96 physical type data into a no-timezone datetime. #59514

hubgeter wants to merge 2 commits into from
+212 −208

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Dec 31, 2025

What problem does this PR solve?

Problem Summary:
This PR fixes the semantics of reading Parquet INT96 values. Previously, INT96 was interpreted as a time zone–aware type, causing the read results to vary with the set timezone configuration.
This PR changes the behavior to always convert INT96 values to UTC, treating them as time zone–independent timestamps.

Release note

Parse Parquet INT96 physical type data into a no-timezone datetime.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 31, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

hubgeter commented Dec 31, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 31, 2025

TPC-H: Total hot run time: 31901 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

------ Round 1 ----------------------------------
q1	17636	4216	4042	4042
q2	2031	366	237	237
q3	10185	1270	734	734
q4	10204	898	316	316
q5	7553	2159	1915	1915
q6	194	171	137	137
q7	946	807	668	668
q8	9283	1430	1150	1150
q9	5510	4736	4721	4721
q10	6815	1810	1428	1428
q11	488	303	290	290
q12	736	742	602	602
q13	17826	3819	3065	3065
q14	296	287	274	274
q15	594	515	518	515
q16	707	689	644	644
q17	720	749	617	617
q18	6650	6505	6381	6381
q19	1112	961	618	618
q20	402	373	247	247
q21	3138	2562	2356	2356
q22	1102	1055	944	944
Total cold run time: 104128 ms
Total hot run time: 31901 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4175	4062	4042	4042
q2	338	395	333	333
q3	2128	2589	2222	2222
q4	1324	1749	1297	1297
q5	4078	4014	3992	3992
q6	208	170	129	129
q7	1890	1845	2102	1845
q8	2610	2464	2361	2361
q9	7224	7161	7124	7124
q10	2499	2784	2338	2338
q11	529	486	458	458
q12	730	769	622	622
q13	3622	4045	3391	3391
q14	406	301	291	291
q15	542	500	498	498
q16	660	657	649	649
q17	1138	1344	1347	1344
q18	7858	7934	8137	7934
q19	920	911	932	911
q20	2095	2018	1911	1911
q21	4697	4477	4237	4237
q22	1139	1086	988	988
Total cold run time: 50810 ms
Total hot run time: 48917 ms

@doris-robot
Copy link

doris-robot commented Dec 31, 2025

TPC-DS: Total hot run time: 174160 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

query5	4490	593	449	449
query6	332	227	213	213
query7	4224	478	279	279
query8	351	254	242	242
query9	8758	2600	2637	2600
query10	492	379	318	318
query11	15178	15043	15060	15043
query12	181	118	115	115
query13	1272	505	416	416
query14	6216	2957	2733	2733
query14_1	2631	2641	2678	2641
query15	198	195	179	179
query16	984	403	469	403
query17	1123	736	570	570
query18	2530	425	333	333
query19	225	221	185	185
query20	117	118	112	112
query21	224	151	115	115
query22	3951	4002	3824	3824
query23	15981	15694	15438	15438
query23_1	15435	15568	15500	15500
query24	7366	1579	1238	1238
query24_1	1216	1201	1181	1181
query25	528	469	401	401
query26	1228	262	157	157
query27	2779	457	291	291
query28	4549	2201	2194	2194
query29	776	523	429	429
query30	313	236	214	214
query31	803	627	556	556
query32	78	71	69	69
query33	528	326	290	290
query34	891	882	540	540
query35	755	801	697	697
query36	903	887	853	853
query37	127	99	78	78
query38	2725	2679	2618	2618
query39	778	743	730	730
query39_1	733	714	710	710
query40	217	138	121	121
query41	73	68	66	66
query42	104	105	107	105
query43	457	449	421	421
query44	1339	762	765	762
query45	191	186	181	181
query46	872	964	617	617
query47	1447	1530	1378	1378
query48	334	337	257	257
query49	647	429	340	340
query50	663	295	232	232
query51	3754	3819	3804	3804
query52	113	112	100	100
query53	300	333	277	277
query54	305	289	260	260
query55	81	81	76	76
query56	302	318	317	317
query57	1017	1064	978	978
query58	270	268	262	262
query59	2092	2141	2172	2141
query60	336	328	302	302
query61	192	181	182	181
query62	383	375	324	324
query63	304	268	270	268
query64	5128	1360	995	995
query65	3761	3688	3768	3688
query66	1433	438	311	311
query67	14934	15281	15815	15281
query68	8237	1010	725	725
query69	510	339	313	313
query70	1048	903	941	903
query71	355	304	279	279
query72	6130	3413	3508	3413
query73	767	723	316	316
query74	8810	8796	8645	8645
query75	2835	2845	2499	2499
query76	3889	1066	662	662
query77	546	386	277	277
query78	9865	9997	9080	9080
query79	1214	929	618	618
query80	669	585	519	519
query81	494	264	230	230
query82	212	145	107	107
query83	263	248	248	248
query84	258	120	109	109
query85	882	525	462	462
query86	319	315	316	315
query87	2872	2858	2758	2758
query88	3197	2324	2291	2291
query89	404	356	332	332
query90	1960	154	159	154
query91	179	164	155	155
query92	70	65	61	61
query93	1008	919	573	573
query94	568	335	289	289
query95	586	344	307	307
query96	588	448	205	205
query97	2310	2381	2291	2291
query98	220	196	196	196
query99	600	577	513	513
Total cold run time: 253913 ms
Total hot run time: 174160 ms

@doris-robot
Copy link

doris-robot commented Dec 31, 2025

ClickBench: Total hot run time: 27.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

query1	0.06	0.05	0.05
query2	0.11	0.06	0.05
query3	0.25	0.10	0.09
query4	1.61	0.11	0.11
query5	0.27	0.24	0.26
query6	1.15	0.64	0.66
query7	0.04	0.02	0.03
query8	0.05	0.04	0.04
query9	0.58	0.51	0.49
query10	0.57	0.55	0.56
query11	0.16	0.11	0.11
query12	0.16	0.13	0.13
query13	0.63	0.60	0.59
query14	0.97	0.99	0.98
query15	0.82	0.80	0.78
query16	0.42	0.42	0.37
query17	1.03	1.05	1.08
query18	0.22	0.21	0.21
query19	1.95	1.91	1.89
query20	0.02	0.02	0.01
query21	15.45	0.30	0.14
query22	4.86	0.05	0.05
query23	16.18	0.30	0.10
query24	0.94	0.61	0.40
query25	0.11	0.08	0.08
query26	0.13	0.14	0.12
query27	0.06	0.06	0.04
query28	4.97	1.05	0.88
query29	12.60	3.97	3.18
query30	0.28	0.13	0.12
query31	2.81	0.67	0.38
query32	3.23	0.57	0.48
query33	2.97	3.03	3.08
query34	16.64	5.09	4.48
query35	4.45	4.48	4.45
query36	0.65	0.50	0.49
query37	0.11	0.06	0.06
query38	0.08	0.04	0.04
query39	0.04	0.04	0.03
query40	0.19	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 97.98 s
Total hot run time: 27.17 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 31, 2025

BE UT Coverage Report

Increment line coverage 78.05% (32/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.39% (18964/35518)
Line Coverage 39.26% (175969/448170)
Region Coverage 33.82% (136138/402479)
Branch Coverage 34.76% (58796/169164)

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 31, 2025

BE Regression && UT Coverage Report

Increment line coverage 78.05% (32/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.15% (24357/34723)
Line Coverage 56.51% (252588/446978)
Region Coverage 51.06% (207627/406642)
Branch Coverage 52.49% (89083/169726)

@hubgeter
Copy link
Contributor Author

hubgeter commented Jan 4, 2026

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 4, 2026

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Jan 4, 2026

TPC-H: Total hot run time: 32002 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ea91c2dbc71d1824222d0ca9cf3c6844d48a0504, data reload: false

------ Round 1 ----------------------------------
q1	17644	4265	4063	4063
q2	2036	360	246	246
q3	10153	1287	706	706
q4	10207	809	312	312
q5	7492	2091	1863	1863
q6	190	171	138	138
q7	953	776	656	656
q8	9289	1431	1167	1167
q9	5207	4509	4498	4498
q10	6809	1817	1390	1390
q11	518	290	286	286
q12	768	742	582	582
q13	17796	3780	3100	3100
q14	290	287	261	261
q15	596	514	512	512
q16	674	692	609	609
q17	667	789	569	569
q18	6982	6578	6763	6578
q19	1173	998	624	624
q20	422	395	272	272
q21	3223	2590	2520	2520
q22	1135	1089	1050	1050
Total cold run time: 104224 ms
Total hot run time: 32002 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4350	4306	4220	4220
q2	356	425	317	317
q3	2197	2785	2474	2474
q4	1389	1844	1401	1401
q5	4640	4377	4387	4377
q6	213	163	127	127
q7	1976	1878	1789	1789
q8	2705	2424	2361	2361
q9	7056	7279	7304	7279
q10	2533	2776	2290	2290
q11	545	459	434	434
q12	669	744	546	546
q13	3269	3753	3060	3060
q14	278	292	256	256
q15	527	491	486	486
q16	628	631	602	602
q17	1092	1301	1331	1301
q18	7380	7322	7355	7322
q19	817	778	818	778
q20	1877	2078	1834	1834
q21	4500	4415	4077	4077
q22	1105	1048	964	964
Total cold run time: 50102 ms
Total hot run time: 48295 ms

@doris-robot
Copy link

doris-robot commented Jan 4, 2026

TPC-DS: Total hot run time: 172457 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ea91c2dbc71d1824222d0ca9cf3c6844d48a0504, data reload: false

query5	4976	585	452	452
query6	320	220	217	217
query7	4210	445	258	258
query8	326	232	245	232
query9	8782	2599	2629	2599
query10	535	373	320	320
query11	15294	15289	14969	14969
query12	175	121	113	113
query13	1269	485	386	386
query14	6469	2962	2663	2663
query14_1	2610	2588	2630	2588
query15	208	192	175	175
query16	994	491	480	480
query17	1078	689	570	570
query18	2617	439	338	338
query19	228	225	196	196
query20	124	117	114	114
query21	213	141	125	125
query22	4176	4004	4100	4004
query23	15939	15574	15443	15443
query23_1	15408	15430	15551	15430
query24	7369	1519	1141	1141
query24_1	1180	1151	1190	1151
query25	558	490	431	431
query26	1234	278	160	160
query27	2747	448	294	294
query28	4582	2118	2124	2118
query29	787	552	447	447
query30	316	239	212	212
query31	804	624	561	561
query32	82	68	69	68
query33	587	326	266	266
query34	900	857	511	511
query35	748	782	691	691
query36	847	885	834	834
query37	121	91	79	79
query38	2702	2717	2649	2649
query39	780	752	733	733
query39_1	714	708	719	708
query40	216	130	116	116
query41	66	60	64	60
query42	105	101	96	96
query43	430	471	405	405
query44	1309	728	725	725
query45	186	183	178	178
query46	846	945	579	579
query47	1451	1470	1364	1364
query48	306	319	232	232
query49	605	419	326	326
query50	626	266	203	203
query51	3792	3756	3807	3756
query52	107	107	96	96
query53	294	327	261	261
query54	278	295	248	248
query55	78	71	69	69
query56	294	291	283	283
query57	1070	985	926	926
query58	264	242	245	242
query59	2068	2101	2030	2030
query60	318	309	291	291
query61	159	166	158	158
query62	391	372	320	320
query63	300	264	267	264
query64	4879	1297	988	988
query65	3760	3704	3710	3704
query66	1391	415	307	307
query67	15359	15363	15134	15134
query68	6223	990	705	705
query69	486	343	307	307
query70	1002	945	868	868
query71	360	298	278	278
query72	6085	3442	3392	3392
query73	761	730	307	307
query74	8814	8755	8584	8584
query75	2792	2814	2477	2477
query76	3908	1036	653	653
query77	516	377	273	273
query78	9853	9743	9152	9152
query79	1604	844	592	592
query80	645	566	463	463
query81	508	260	225	225
query82	212	142	110	110
query83	253	255	235	235
query84	260	128	94	94
query85	894	510	459	459
query86	381	328	324	324
query87	2840	2852	2749	2749
query88	3191	2232	2218	2218
query89	374	349	326	326
query90	2170	146	143	143
query91	170	164	140	140
query92	83	63	63	63
query93	1615	900	525	525
query94	570	316	293	293
query95	579	374	306	306
query96	585	458	201	201
query97	2299	2351	2272	2272
query98	228	209	195	195
query99	584	593	523	523
Total cold run time: 253740 ms
Total hot run time: 172457 ms

@doris-robot
Copy link

doris-robot commented Jan 4, 2026

ClickBench: Total hot run time: 26.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ea91c2dbc71d1824222d0ca9cf3c6844d48a0504, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.05
query3	0.27	0.09	0.08
query4	1.61	0.11	0.10
query5	0.27	0.27	0.26
query6	1.16	0.66	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.55	0.56	0.55
query11	0.15	0.10	0.09
query12	0.13	0.10	0.10
query13	0.61	0.58	0.59
query14	0.94	0.95	0.95
query15	0.80	0.80	0.81
query16	0.39	0.40	0.40
query17	1.08	1.05	1.01
query18	0.23	0.22	0.21
query19	1.93	1.85	1.84
query20	0.02	0.01	0.01
query21	15.40	0.27	0.14
query22	5.24	0.05	0.05
query23	16.04	0.30	0.10
query24	1.43	0.35	0.35
query25	0.08	0.07	0.07
query26	0.14	0.13	0.13
query27	0.09	0.05	0.08
query28	3.69	1.06	0.88
query29	12.53	3.85	3.13
query30	0.28	0.13	0.12
query31	2.83	0.65	0.40
query32	3.25	0.57	0.46
query33	3.01	3.06	3.07
query34	16.73	5.11	4.41
query35	4.49	4.46	4.44
query36	0.63	0.50	0.49
query37	0.12	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.35 s
Total hot run time: 26.9 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 4, 2026

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

dev/4.0.x kind/behavior-changed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hubgeter @Thearas @doris-robot @hello-stephen @morningman