Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Weird subtitle offset when the start time of input video is not 0 #205

Closed
AceCLee opened this issue Feb 24, 2020 · 8 comments
Closed

Weird subtitle offset when the start time of input video is not 0 #205

AceCLee opened this issue Feb 24, 2020 · 8 comments

Comments

@AceCLee
Copy link

AceCLee commented Feb 24, 2020 •
edited
Loading

Hello Rigaya.

part of log:

CheckGPUListByEncoder: Success.
GPUAutoSelect: Success.
avhw reader selected.
InitInput: input selected : 9.
Probing matroska,webm score:100 size:2048
[matroska,webm @ 000001feba2390c0] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 1000000 from timebase
avcuvid/avsw: opened file "test.mkv".
[matroska,webm @ 000001feba2390c0] Before avformat_find_stream_info() pos: 5480 bytes read:8584 seeks:2 nb_streams:1
[h264 @ 000001feba23d700] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001feba23d700] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 000001feba23d700] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001feba23d700] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 000001feba23d700] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001feba23d700] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 000001feba23d700] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 000001feba23d700] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 000001feba23d700] Format yuv420p chosen by get_format().
[h264 @ 000001feba23d700] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 000001feba23d700] no picture
[matroska,webm @ 000001feba2390c0] All info found
[matroska,webm @ 000001feba2390c0] stream 0: start_time: 0.042 duration: -9223372036854776.000
[matroska,webm @ 000001feba2390c0] format: start_time: 0.042 duration: 1401.942 (estimate from stream) bitrate=7378 kb/s
[matroska,webm @ 000001feba2390c0] After avformat_find_stream_info() pos: 163800 bytes read:166904 seeks:2 frames:4
avcuvid/avsw: got stream information.
Input #0, matroska,webm, from 'test.mkv':
Metadata:
encoder : libebml v1.3.10 + libmatroska v1.5.2
creation_time : 2020-02-24T09:11:17.000000Z
Duration: 00:23:21.94, start: 0.042000, bitrate: 7378 kb/s
Stream #0:0, 4, 1/1000: Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)
Metadata:
BPS-eng : 7377326
DURATION-eng : 00:23:21.942000000
NUMBER_OF_FRAMES-eng: 33613
NUMBER_OF_BYTES-eng: 1292823004
_STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 64-bit
_STATISTICS_WRITING_DATE_UTC-eng: 2020-02-24 09:11:17
_STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
avcuvid/avsw: found video stream, stream idx 0
avcuvid/avsw: use video stream #0 for input, codec h264, stream time_base 1/1000, codec_timebase 1001/48000.
avcuvid/avsw: can be decoded by cuvid.
avcuvid: initialized h264_mp4toannexb filter.
avcuvid: start predecode.
avcuvid: GetHeader: changed 41 bytes -> 38 bytes by h264_mp4toannexb.
avcuvid: GetHeader: 38 bytes.
avcuvid: initialized h264 codec context for parser: time_base: 1001/48000, pkt_timebase: 1/1000.
avcuvid: fps decoder invalid: false
[h264 @ 000001fed37dd040] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001fed37dd040] nal_unit_type: 8(PPS), nal_ref_idc: 3
avcuvid: found first key frame: timestamp 42 (0.042), offset 0
avcuvid: read 128 packets.
avcuvid: checking 111 frame samples.
avcuvid: stream timebase 1001/48000
avcuvid: decoder fps 24000/1001
avcuvid: duration histgram of 2 frames
avcuvid: 42 [ 79 frames]
avcuvid: 41 [ 32 frames]
avcuvid: read 256 packets.
avcuvid: checking 239 frame samples.
avcuvid: stream timebase 1001/48000
avcuvid: decoder fps 24000/1001
avcuvid: duration histgram of 2 frames
avcuvid: 42 [169 frames]
avcuvid: 41 [ 70 frames]
avcuvid: fps mul: 1001
avcuvid: raw avgDuration: 41.707113
avcuvid: estimatedAvgFps: 24001/1001
avcuvid: use decoder fps...
avcuvid: final AvgFps (raw64): 24001/1001
avcuvid: final AvgFps (gcd): 24000/1001
avcuvid: final AvgFps (round): 24000/1001
avcuvid: avcuvid: H.264/AVC, 1920x1080, 24000/1001 fps
avcuvid: chromaloc:left
avcuvid: sar 1:1, shift 0
timestamp check: 0x1
InitInput: Success.
InitDevice: device #0 (GeForce RTX 2060) selected.
InitNVEncInstance: Success.
cuvid: nvcuvid.dll available
cuvid: created frame queue
cuvid: created video parser
cuvid: DecodePacket: success
InitDecoder: Success.
cspconv: cspconv(nv12 -> yv12)
subburn: trying to open subtitle file "test.ass"
[NULL @ 000001fed4f53140] Opening 'test.ass' for reading
[file @ 000001fecab213c0] Setting default whitelist 'file,crypto'
Probing ass score:100 size:2048
Probing lrc score:5 size:2048
[ass @ 000001fed4f53140] Format ass probed with size=2048 and score=100
[ass @ 000001fed4f53140] Before avformat_find_stream_info() pos: 37169 bytes read:37169 seeks:0 nb_streams:1
[ass @ 000001fed4f53140] All info found
[ass @ 000001fed4f53140] stream 0: start_time: -92233720368547760.000 duration: -92233720368547760.000
[ass @ 000001fed4f53140] format: start_time: -9223372036854.775 duration: -9223372036854.775 (estimate from bit rate) bitrate=0 kb/s
[ass @ 000001fed4f53140] After avformat_find_stream_info() pos: 37169 bytes read:37169 seeks:0 frames:0
subburn: got stream information.
Input #0, ass, from 'test.ass':
Duration: N/A, bitrate: N/A
Stream #0:0, 0, 1/100: Subtitle: ass
subburn: found subtitle in stream #0 (ass), timebase 1/100.
subburn: sub type: text
subburn: set "sub_charenc" to "UTF-8"
subburn: set "sub_text_format" to "ass"
subburn: Subtitle Decoder opened
subburn: Subtitle Decode Info: ass, 0x0
Raster: FreeType 2.7.1
Shaper: FriBidi 1.0.1 (SIMPLE)
Initialized

subburn: decoded subtitle chunk (6.542 - 10.552), Video frame (0)
Event at 6542, +4010: 0,0,staff,,0,0,0,,{\fad(500,500)}
subburn: decoded subtitle chunk (10.552 - 14.552), Video frame (0)
Event at 10552, +4000: 1,0,staff,,0,0,0,,{\fad(500,500)}

original subtitle:

Dialogue: 0,0:00:06.98,0:00:10.99,staff,,0,0,0,,{\fad(500,500)}
Dialogue: 0,0:00:10.99,0:00:14.99,staff,,0,0,0,,{\fad(500,500)}

first key frame: timestamp 42 (0.042)

offset of all subtitles: 438ms

But I can get correct time stamp info with a video whose start time is 0.

vidInputOffsetMs
in

const int64_t nStartTime = av_rescale_q(m_subData->pts, av_make_q(1, AV_TIME_BASE), av_make_q(1, 1000)) - vidInputOffsetMs + tsOffsetMs;

is 0 when start time of video is 0, but when start time is not 0, vidInputOffsetMs may be an incorrect value.

const int64_t vidInputOffsetMs = av_rescale_q(prm->videoInputFirstKeyPts, prm->videoTimebase, { 1, 1000 });

in NVEnc/NVEncFilterSubburn.cpp#L441
sets the timebase 1/1000.

part of log of subburn:

subburn: found subtitle in stream #0 (ass), timebase 1/100.

Maybe there are something wrong in

const int64_t vidInputOffsetMs = av_rescale_q(prm->videoInputFirstKeyPts, prm->videoTimebase, { 1, 1000 });

Making subtitle with Aegisub needs a whole video file (with audio), but it seems like aegisub will treat the start time of video as start time of subtitle, the time stamp info of subtitle is consistent with the time stamp of video stream. Maybe there is no need to minus vidInputOffsetMs in time stamp of subburn when format of subtitle is ass.
@rigaya
Copy link
Owner

rigaya commented Feb 24, 2020

Thank you for pointing out and debugging the issue!

As you have mentioned, I also think the line below is wrong.

const int64_t vidInputOffsetMs = av_rescale_q(prm->videoInputFirstKeyPts, prm->videoTimebase, { 1, 1000 });

In that case, prm->videoTimebase is 1001/96000, but actually we need to use 1/1000 for that calculation as

avcuvid/avsw: use video stream #0 for input, codec h264, stream time_base 1/1000,

This is causing the offset which should be "42ms" to be the wrong value "438ms". I'll think for a way to fix this issue.

Making subtitle with Aegisub needs a whole video file (with audio), but it seems like aegisub will treat the start time of video as start time of subtitle, the time stamp info of subtitle is consistent with the time stamp of video stream. Maybe there is no need to minus vidInputOffsetMs in time stamp of subburn when format of subtitle is ass.

It's difficult to choose which behavior is better when you add subtitles from a separate file, as I had a case in the past that the offset was required.

@AceCLee
Copy link
Author

AceCLee commented Feb 25, 2020

Thanks for reply.
Is it possible to add a parameter of vpp subburn to control whether subburn enables offset?

@Abu3safeer
Copy link

Did you add subtitle directly to the encoder? or use avisynth to burn it on the video?

@AceCLee
Copy link
Author

AceCLee commented Feb 26, 2020

Did you add subtitle directly to the encoder? or use avisynth to burn it on the video?

I added subtitle to vpp subburn of nvenc.

@Abu3safeer
Copy link

Sorry, but I have no experience is this, I use avisynth to burn ass subtitle (since it uses vsfilter).

@rigaya
Copy link
Owner

rigaya commented Feb 26, 2020

@AceCLee

Is it possible to add a parameter of vpp subburn to control whether subburn enables offset?

Sure, I'll add it in the next release.

rigaya added a commit that referenced this issue Feb 29, 2020
@rigaya
vpp-subburnで、動画のtimestampが0始まりでなかった場合の時刻調整の誤りを修正。( #205 )
2537df2 
入力のtimebaseでなく、出力のtimebaseを用いていた。
@rigaya
Copy link
Owner

rigaya commented Feb 29, 2020

I have updated to NVEnc 4.66, which has fixed the wrong timestamp offset calculation. It also has new parameter for vpp-subburn, you should be able to disable timestamp offset by
--vpp-subburn filename="sub file",vid_ts_offset=off.

@AceCLee
Copy link
Author

AceCLee commented Feb 29, 2020

Thanks for your efforts!

@AceCLee AceCLee closed this as completed Feb 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rigaya @Abu3safeer @AceCLee