Did some additional testing, with CQP and even tried h264 and they all yield the same results. It just freezes on these specific frames.

Piece of data I forgot yesterday is that I am using NVIDIA driver 399.07.

Logs provided indicates nothing strage happening, I cannot say what is causing the problem here. If this depends on the input, it might be diffcult to solve.

Would you please test for raw output without muxing (output to ".265" extention), to identify whether muxing is causing the problem or not?

Thanks for responding!

I was previously using raw .h265 output with the same issue (but was using mkvtoolkit to mux it into a MKV file). I can do that again with logs if you like.

I am trying to think of a way to get you a piece of the source file to test or the produced MKV (if that would even help).

Could clock speeds or something affect this? Are there other logs that might help?

Thanks for sharing information about raw output, I was thinking of the case that the muxing procedure by NVEncC was causing the issue, but if muxing by mkvtoolkit results the same, it is not the case.

Also, as you have run into the issue with both avsw & avhw, decoding shall not be the problem either.

It seems to be a problem on encode stage, but I currently have no idea...

About the clock speed, I don't think clock speed is affecting this. If you have changed the clock speed from default, there is a possibility that it might cause some problem, but it wouldn't be like that the problem causes always at the same scene, it will rather be more random.

Is there a way to get more logging or something to show the issue. To describe it a little more, it seems like nvenc is just encoding the same frame over and over again because when I jump around in VLC, when I reach one of these stuck frames, it shows the same stuck frame whenever i jump into a timestamp with the stuck frame issue. So it appears that it is actually being encoded that way rather than missing data or the raw stream. Also, plex playback shows the same thing.

If i try to encode with the same settings in FFMPEG CPU, it encodes at .4fps which is really too long to be a realistic option. I really want this to work. On another note, I will test a different NVIDIA card over the weekend to see if it improves the results.

just encoding the same frame over and over again

May be checking timestamps of the frames might be a hint, would you please test adding the options below, and share the output logs?

The options below will output the frame infomation from demuxer & muxer of NVEncC.

--trim 0:35000 --log-framelist "output.demuxlog.txt" --log-mux-ts "output.muxlog.txt"

I will give it a shot as soon I can. Thanks so much for the help here!

Here are the logs... Freeze occurs at 00:20:47 into the playback.
output.muxlog2.txt
output.demuxlog2.txt

Let me know what you find. I will continue to tinker with the command to see if there is an option that is affecting it.

Thanks for providing the logs.

Even though the timestamps has no problems, it seems that the same frame is encoded over and over again, starting arround 00:20:47, as you have said.

Although I currently have no idea what is causing the issue (and why only from that specific scene it gets stuck), I'll try think of a way to debug further.

Would you please try following commands, to check if decoding process of NVEncC is fine or not. (Below will use ffmpeg sw decoder)

ffmpeg.exe -y -i input.mkv -t 35000 -strict -1 -an -f yuv4mpegpipe - | NVEncC64.exe --y4m -i - -c hevc --output-depth 10 --log-level debug --log-mux-ts "output.muxlog.txt" -o output.mkv > output.y4m.log.txt 2>&1

If this is fine, the issue is caused by decoder of NVEncC, otherwise the problem is caused somewhere during encode.

Thanks for the feedback! I just tried your command and the resulting MKV file works fine, no freezing. So the problem must lie in the NVEncC decoder.

Not sure they would help, but I have attached the logs from that run. Also, just to note, there are an additional 2-3 places in the movie where the freezing happens, not just this one scene, I have just been focusing on this one since its the first.

Let me know if there is anything else i can test. For now, I will try to adapt your piped command here to see if I can get a proper 4k HDR MKV. I may even be able to use FFMPEG's HW decode.

Thanks again for all the help.

logs.zip

I was able to use the pipe method you showed me above which allowed me to encode at 20fps which is an improvement and from what I can tell, it worked great.

Let me know if you want me to test anything else or if you build a debug binary or something that will produce the logging you need.

Thanks for your time so far.

It's nice to hear that the command worked out fine, thank you for all the tests.

However, I'm not been able to reproduce the issue in deocde of NVEncC (actually might be hard without input file...)

I'll try debugging with the logs.

The original source file is 55GB so it would be hard to get it to you. However, I used ffmpeg with -ss and -t to produce a 30 second clip, still ~170MB, that when encoded with NVencC will produce the frozen frames. I have sent a link to download it to your email address listed in github.

It actually happens twice in this clip, once at 4 seconds and once at 18 seconds.

Here is the command I have been using to encode:

NVEncC64.exe --avhw cuda --sar 1:1 --vbrhq 29000 --codec h265 --preset quality --output-depth 10 --level 5.1 --profile main10 --tier high --gop-len 12 --lookahead 32 --no-b-adapt --no-i-adapt --qp-init 1 --vbr-quality 17 --aq --max-bitrate 29000 --ref 1 --colormatrix bt2020nc --colorprim bt2020 --transfer smpte2084 --chromaloc 2 --videoformat ntsc --mv-precision Q-pel --master-display G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,50) --crop 0,276,0,276 -i "D:\trim2.mkv" -o "D:\trim2-nvencc.mkv"

Hopefully this helps you do some local troubleshooting. Thanks again!

Thank you so much for sending the sample file, I was able to reproduce the issue. Debugging localy with the file is helping me very much.

I've now got some hints by debugging with the file, but have not figured out how to fix yet, probably will need some time to fix.

I have updated to NVEnc 4.15, fixing this issue with avsw. (But unfortunately, not for avhw)

For avhw, I'm actually getting wierd result.

The commands below works fine, even with avhw.

mkvextract.exe input.mkv tracks 0:temp.hevc
NVEncC64.exe --avhw -c hevc --output-depth 10 -i temp.hevc -o output.mkv

The next command should also do the same, but there is the issue.

ffmpeg.exe -i input.mkv -an -c:v copy -bsf:v hevc_mp4toannexb temp.hevc
NVEncC64.exe --avhw -c hevc --output-depth 10 -i temp.hevc -o output.mkv

I'll debug further.

Thanks so much for the work on AVSW. It's working great at ~50fps. Keep me posted on AVHW.

Thanks again!

Now NVEnc 4.16 can decode properly with avhw.

Thank you again for all the tests and samples.

Thank you! I will give it a test and let you know!

@rigaya 4.16 is working great. Thanks for your support! Awesome tool.