The auto max bitrate sets 96% of the value of the spec of main profile.

12000 x 0.96 = 11520
25000 x 0.96 = 24000

This is because there was a case that the NVENC SDK returned error ("8 (NVENC indicates that one or more of the parameter passed to the API call is invalid.)") when you specified the exact value of the spec. I don't know why but setting to 96% works fine.

main10 / main444 profile is not considered here.

Thanks for reply!

input video: 1080p yuv420p10 24000/1001
cmd param: --codec hevc --ref 16 --level auto --vbr/--vbrhq
nvenc cmd output: (Max: 11520 kbps)
output video level: 5

In this case, correct level should be 5 because ref is 16, but nvenc specified 11520kbps as max bitrate, it should be 24000kbps as I exactly specified --level 5 in cmd param.

And I can specify an exact value for max bitrate in nvencc 4.65:
--codec hevc --vbrhq xxx --max-bitrate 25000 --level 5 --profile auto --tier main --ref 16

In this case, correct level should be 5 because ref is 16, but nvenc specified 11520kbps as max bitrate, it should be 24000kbps as I exactly specified --level 5 in cmd param.

It is not checking ref frames.

And I can specify an exact value for max bitrate in nvencc 4.65:
--codec hevc --vbrhq xxx --max-bitrate 25000 --level 5 --profile auto --tier main --ref 16

There was some cases (not always) that caused error in the past. If you really care the few percent, then please specify the exact value.

OK. Maybe nvencc should take --ref into consideration when specify --level auto.

The updated NVEnc 4.66 will have --ref value considered when calculating max bitrate automatically.

Thanks for your efforts! It works!