Bug#16997513 MY_STRTOLL10 ACCEPTING OVERFLOWED UNSIGNED LONG LONG VALUES AS NORMAL ONES

Joao Gramacho · Joao Gramacho · commit be9dcdf96050 · 2013-07-31T17:54:40.000+01:00
Problem:
=======
It was detected an incorrect behavior of my_strtoll10 function when 
converting strings with numbers in the following format:
"184467440XXXXXXXXXYY"

Where XXXXXXXXX &gt; 737095516 and YY &lt;= 15

Samples of problematic numbers:
"18446744073709551915"
"18446744073709552001"

Instead of returning the larger unsigned long long value and setting overflow
in the returned error code, my_strtoll10 function returns the lower 64-bits 
of the evaluated number and did not set overflow in the returned error code.

Analysis:
========
Once trying to fix bug 16820156, I've found this bug in the overflow check of
my_strtoll10 function.

This function, once receiving a string with an integer number larger than
18446744073709551615 (the larger unsigned long long number) should return the
larger unsigned long long number and set overflow in the returned error code.

Because of a wrong overflow evaluation, the function didn't catch the
overflow cases where (i == cutoff) &amp;&amp; (j &gt; cutoff2) &amp;&amp; (k &lt;= cutoff3). When
the overflow evaluation fails, the function return the lower 64-bits of the
evaluated number and do not set overflow in the returned error code.

Fix:
===
Corrected the overflow evaluation in my_strtoll10.
diff --git a/strings/my_strtoll10.c b/strings/my_strtoll10.c
@@ -206,8 +206,8 @@ longlong my_strtoll10(const char *nptr, char **endptr, int *error)
     goto overflow;
 
   /* Check that we didn't get an overflow with the last digit */
-  if (i > cutoff || (i == cutoff && ((j > cutoff2 || j == cutoff2) &&
-                                     k > cutoff3)))
+  if (i > cutoff || (i == cutoff && (j > cutoff2 || (j == cutoff2 &&
+                                     k > cutoff3))))
     goto overflow;
   li=i*LFACTOR2+ (ulonglong) j*100 + k;
   return (longlong) li;
