avcodec/osq: Fix signed integer overflow in update_stats()

Fixes: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: 410109093/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_OSQ_fuzzer-6550900028276736

Note, none of the available osq files uses update_stats(), this change may fix or break
files using coding_mode == 2. The code prior looks wrong though

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <[email protected]>
(cherry picked from commit c909ef31be96b6983698c3b01c675de8e5f2637a)
Signed-off-by: Michael Niedermayer <[email protected]>

diff --git a/libavcodec/osq.c b/libavcodec/osq.c
index 99a035c5c53e2c7fef7ba828c6b2ce92d1653895..256b1d64215b5e4086e7df07e3fa6232191543c7 100644 (file)
--- a/libavcodec/osq.c
+++ b/libavcodec/osq.c
@@ -145,8 +145,8 @@ static void reset_stats(OSQChannel *cb)
 
 static void update_stats(OSQChannel *cb, int val)
 {
-    cb->sum += FFABS(val) - cb->history[cb->pos];
-    cb->history[cb->pos] = FFABS(val);
+    cb->sum += FFABS((int64_t)val) - cb->history[cb->pos];
+    cb->history[cb->pos] = FFABS((int64_t)val);
     cb->pos++;
     cb->count++;
     if (cb->pos >= FF_ARRAY_ELEMS(cb->history))