Support AV1 hardware acceleration on Apple M3 and newer #6339

Nomis101 · 2024-10-06T16:22:24Z

Will close #5487
This patch did not made it into FFmpeg 7.1 FFmpeg/FFmpeg@f9c5c53

I do not have a Mac with M3, so I can not test (will get an M4 when available, then I could test this).

galad87 · 2024-10-08T07:14:04Z

Neither do I, it would be better to test it before merging it.

bradleysepos · 2024-10-08T14:13:39Z

@galad87 Any chance you can link a build for testing?

galad87 · 2024-10-08T14:15:13Z

There are the GitHub CI builds that can be accessed when logged into GitHub, here's the one for this PR: https://github.com/HandBrake/HandBrake/actions/runs/11204107074?pr=6339

bradleysepos · 2024-10-08T15:36:09Z

Ah, right!

bradleysepos · 2024-10-08T15:38:20Z

@mstorsjo If you feel like testing this one, build linked above (likely need to be logged into GitHub for access). VT decoding can be enabled in preferences, and the activity log for any encode should indicate whether it is used.

mstorsjo · 2024-10-08T20:54:17Z

I tested this, but I'm actually not quite sure that it does get activated.

Here's the activity log from a test transcode:

[22:47:47] macgui: Handbrake Version: 20241006181301-f5699bf (2024100601)
[22:47:47] Compile-time hardening features are enabled
[22:47:47] macgui: built-in presets updated
[22:47:47] hb_init: starting libhb thread
[22:47:47] macgui: Error domain: NSCocoaErrorDomain
[22:47:47] macgui: Error code: 259
[22:47:47] macgui: Error description: The file couldn’t be opened because it isn’t in the correct format.
[22:47:47] macgui: Error debug description: Error Domain=NSCocoaErrorDomain Code=259 "The file couldn’t be opened because it isn’t in the correct format."
[22:48:44] macgui: ScanCore scanning titles with a duration of 10 seconds or more
[22:48:44] CPU: Apple M3 Pro
[22:48:44]  - logical processor count: 11
[22:48:44] hb_scan: path=/Users/martin/Movies/av1/Chimera-AV1-8bit-1920x1080-6736kbps.mp4, title_index=0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/martin/Movies/av1/Chimera-AV1-8bit-1920x1080-6736kbps.mp4':
  Metadata:
    major_brand     : iso4
    minor_version   : 1
    compatible_brands: iso4av01
    creation_time   : 2018-08-31T01:00:56.000000Z
  Duration: 00:06:12.41, start: 0.000000, bitrate: 6734 kb/s
  Stream #0:0[0x1](und): Video: av1 (libdav1d) (Main) (av01 / 0x31307661), yuv420p(tv), 1920x1080, 6733 kb/s, 23.98 fps, 23.98 tbr, 24k tbn (default)
      Metadata:
        creation_time   : 2018-08-31T01:00:56.000000Z
        handler_name    : [email protected]_master
        vendor_id       : [0][0][0][0]
[22:48:44] scan: decoding previews for title 1
[22:48:45] using bitstream PAR 1:1
[22:48:45] scan: 10 previews, 1920x1080, 23.976 fps, autocrop = 34/34/0/0, aspect 16:9, PAR 1:1, color profile: 1-1-1, chroma location: unspecified
[22:48:45] libhb: scan thread found 1 valid title(s)
[22:48:45] macgui: ScanCore scan done
[22:51:59] macgui: Failed to create bookmark: Error Domain=NSCocoaErrorDomain Code=256 "Could not open() the item" UserInfo={NSURL=file:///Users/martin/Library/Containers/fr.handbrake.HandBrake/Data/Movies/, NSDebugDescription=Could not open() the item}
[22:51:59] macgui: Failed to create bookmark: Error Domain=NSCocoaErrorDomain Code=256 "Could not open() the item" UserInfo={NSURL=file:///Users/martin/Library/Containers/fr.handbrake.HandBrake/Data/Movies/, NSDebugDescription=Could not open() the item}
[22:51:59] macgui: Failed to create bookmark: Error Domain=NSCocoaErrorDomain Code=256 "Could not open() the item" UserInfo={NSURL=file:///Users/martin/Library/Containers/fr.handbrake.HandBrake/Data/Movies/, NSDebugDescription=Could not open() the item}
[22:52:04] Compile-time hardening features are enabled
[22:52:04] hb_init: starting libhb thread
[22:52:04] macgui: fr.handbrake.HandBrakeXPCService scanning specifically for title: 1
[22:52:04] CPU: Apple M3 Pro
[22:52:04]  - logical processor count: 11
[22:52:04] hb_scan: path=/Users/martin/Movies/av1/Chimera-AV1-8bit-1920x1080-6736kbps.mp4, title_index=1
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/martin/Movies/av1/Chimera-AV1-8bit-1920x1080-6736kbps.mp4':
  Metadata:
    major_brand     : iso4
    minor_version   : 1
    compatible_brands: iso4av01
    creation_time   : 2018-08-31T01:00:56.000000Z
  Duration: 00:06:12.41, start: 0.000000, bitrate: 6734 kb/s
  Stream #0:0[0x1](und): Video: av1 (libdav1d) (Main) (av01 / 0x31307661), yuv420p(tv), 1920x1080, 6733 kb/s, 23.98 fps, 23.98 tbr, 24k tbn (default)
      Metadata:
        creation_time   : 2018-08-31T01:00:56.000000Z
        handler_name    : [email protected]_master
        vendor_id       : [0][0][0][0]
[22:52:04] scan: decoding previews for title 1
[22:52:04] using bitstream PAR 1:1
[22:52:04] scan: 10 previews, 1920x1080, 23.976 fps, autocrop = 34/34/0/0, aspect 16:9, PAR 1:1, color profile: 1-1-1, chroma location: unspecified
[22:52:04] libhb: scan thread found 1 valid title(s)
[22:52:04] macgui: Failed to create bookmark: Error Domain=NSCocoaErrorDomain Code=256 "Could not open() the item" UserInfo={NSURL=file:///Users/martin/Library/Containers/fr.handbrake.HandBrake/Data/Movies/, NSDebugDescription=Could not open() the item}
[22:52:04] macgui: fr.handbrake.HandBrakeXPCService scan done
[22:52:04] macgui: fr.handbrake.HandBrakeXPCService started encoding Chimera-AV1-8bit-1920x1080-6736kbps.mp4
[22:52:04] macgui: fr.handbrake.HandBrakeXPCService with preset Fast 1080p30 (Modified)
[22:52:04] Starting work at: Tue Oct  8 22:52:04 2024

[22:52:04] 1 job(s) to process
[22:52:04] Starting Task: Encoding Pass
[22:52:04] work: only 1 chapter, disabling chapter markers
[22:52:04] job configuration:
[22:52:04]  * source
[22:52:04]    + /Users/martin/Movies/av1/Chimera-AV1-8bit-1920x1080-6736kbps.mp4
[22:52:04]    + title 1, chapter(s) 1 to 1
[22:52:04]    + container: mov,mp4,m4a,3gp,3g2,mj2
[22:52:04]    + data rate: 6734 kbps
[22:52:04]  * destination
[22:52:04]    + /Users/martin/Library/Containers/fr.handbrake.HandBrake/Data/Movies/Chimera-AV1-8bit-1920x1080-6736kbps.mp4
[22:52:04]    + container: MPEG-4 (libavformat)
[22:52:04]      + align initial A/V stream timestamps
[22:52:04]  * video track
[22:52:04]    + decoder: libdav1d 8-bit (yuv420p)
[22:52:04]      + bitrate 6733 kbps
[22:52:04]    + filters
[22:52:04]      + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[22:52:04]      + Decomb (mode=39)
[22:52:04]      + Framerate Shaper (mode=2:rate=27000000/900000)
[22:52:04]        + frame rate: 23.976 fps -> peak rate limited to 30.000 fps
[22:52:04]      + Crop and Scale (width=1920:height=1012:crop-top=34:crop-bottom=34:crop-left=0:crop-right=0)
[22:52:04]        + source: 1920 * 1080, crop (34/34/0/0): 1920 * 1012, scale: 1920 * 1012
[22:52:04]    + Output geometry
[22:52:04]      + storage dimensions: 1920 x 1012
[22:52:04]      + pixel aspect ratio: 1 : 1
[22:52:04]      + display dimensions: 1920 x 1012
[22:52:04]    + encoder: H.265 (VideoToolbox)
[22:52:04]      + preset:  quality
[22:52:04]      + profile: main
[22:52:04]      + quality: 22.00 (CQ)
[22:52:04]      + color profile: 1-1-1
[22:52:04]      + color range: tv
[22:52:04]      + chroma location: unspecified
[22:52:04] sync: expecting 8928 video frames
[22:52:04] encvt_Init: encoding with constant quality 22.000000
[22:52:04] encvt_Init: com.apple.videotoolbox.videoencoder.ave.hevc
[22:52:04] sync: first pts video is 0
[22:52:04] sync: "Chapter 1" (1) at frame 1 time 0
[22:52:44] reader: done. 1 scr changes
[22:52:45] work: average encoding speed for job is 223.402618 fps
[22:52:45] comb detect: heavy 509 | light 1831 | uncombed 6589 | total 8929
[22:52:45] decomb: deinterlaced 509 | blended 1831 | unfiltered 6589 | total 8929
[22:52:45] vfr: 8929 frames output, 0 dropped and 0 duped for CFR/PFR
[22:52:45] vfr: lost time: 0 (0 frames)
[22:52:45] vfr: gained time: 0 (0 frames) (0 not accounted for)
[22:52:45] libdav1d-decoder done: 8929 frames, 0 decoder errors
[22:52:45] sync: got 8929 frames, 8928 expected
[22:52:45] sync: framerate min 23.976 fps, max 23.976 fps, avg 23.976 fps
[22:52:45] mux: track 0, 8929 frames, 106955958 bytes, 2297.32 kbps, fifo 4096
[22:52:45] Finished work at: Tue Oct  8 22:52:45 2024

[22:52:45] libhb: work result = 0
[22:52:45] macgui: Failed to create bookmark: Error Domain=NSCocoaErrorDomain Code=256 "Could not open() the item" UserInfo={NSURL=file:///Users/martin/Library/Containers/fr.handbrake.HandBrake/Data/Movies/, NSDebugDescription=Could not open() the item}
[22:52:45] macgui: Queue Done, there are no more pending encodes
[22:52:45] macgui: Failed to create bookmark: Error Domain=NSCocoaErrorDomain Code=256 "Could not open() the item" UserInfo={NSURL=file:///Users/martin/Library/Containers/fr.handbrake.HandBrake/Data/Movies/, NSDebugDescription=Could not open() the item}
[22:52:45] macgui: fr.handbrake.HandBrakeXPCService work done

I get near the exact same encoding speed (at around 223 fps) regardless of whether I check or uncheck the "Enable the VideoToolbox hardware decoders" box in settings. The log doesn't really say much about whether it is being used or not, as far as I can see.

AV1 is a bit special in the sense that there's no default built-in decoder in libavcodec, but normally you'd use the libdav1d wrapper instead. But if you do hwaccel decoding, it uses the av1 decoder instead, which is a stub decoder that only does callbacks for hwaccels.

If decoding with ffmpeg -v verbose -hwaccel videotoolbox -i Chimera-AV1-8bit-1920x1080-6736kbps.mp4 -f null -, I see this bit in the log:

Selecting decoder 'av1' because of requested hwaccel method videotoolbox

This comes from fftools/ffmpeg_demux.c in ffmpeg, but it's based on common libavcodec methods - so I'm not entirely sure if this goes the same way when handbrake is calling those APIs or not.

But the log still contains this bit:

  Stream #0:0[0x1](und): Video: av1 (libdav1d) (Main), 1 reference frame (av01 / 0x31307661), yuv420p(tv), 1920x1080, 6733 kb/s, 23.98 fps, 23.98 tbr, 24k tbn (default)

Anyway, the handbrake log above does contain this bit:

[22:52:04]    + decoder: libdav1d 8-bit (yuv420p)

So that makes it seem like it possibly doesn't actually enable the hwaccel at all.

Regarding decoding speed, so with ffmpeg -i Chimera-AV1-8bit-1920x1080-6736kbps.mp4 -f null -, I'm getting 770 fps decoding with libdav1d, but 180 fps when using the videotoolbox hwaccel. So as the transcode above ran at 223 fps, it seems unlikely that it did use the videotoolbox hwaccel decoder.

bradleysepos · 2024-10-08T21:00:03Z

Filters can cause fallback to software decoding. Perhaps try disabling cropping, everything on the filters tab, and set the frame rate to same as source variable.

mstorsjo · 2024-10-08T21:17:18Z

Filters can cause fallback to software decoding. Perhaps try disabling cropping, everything on the filters tab, and set the frame rate to same as source variable.

No notable difference while doing that.

I think we'd want a build with a lot more debug printouts around those relevant areas (around hwaccel specific decoder selection), to be able to say anything about what it's doing.

galad87 · 2024-10-09T04:39:06Z

Hardware encoders are disabled by default, there is an option to enable them in the Settings -> Advanced panel.
Actually there are two options, the first enables them during scan, and when encoding if everything can be run on the GPU, the second options always enables the decoders.

I guess you need the second checkbox too because the Decomb can run only on the CPU, or try one of the VideoToolbox presets that already have all the filters disabled, or maybe it's not actually working? 😅

The activity log will show
[06:34:24] scan: supported video decoders: avcodec hwaccel

if the hardware decoder could be used during the scan, and the job configuration something like

[06:35:23] + decoder: hevc 8-bit hwaccel (nv12, videotoolbox_vld)

when encoding.

mstorsjo · 2024-10-09T05:59:38Z

Hardware encoders are disabled by default, there is an option to enable them in the Settings -> Advanced panel.

I presume you mean decoders, not encoders here?

Actually there are two options, the first enables them during scan, and when encoding if everything can be run on the GPU, the second options always enables the decoders.

In my test above, I did check both boxes.

I guess you need the second checkbox too because the Decomb can run only on the CPU, or try one of the VideoToolbox presets that already have all the filters disabled, or maybe it's not actually working? 😅

The activity log will show [06:34:24] scan: supported video decoders: avcodec hwaccel

if the hardware decoder could be used during the scan, and the job configuration something like

[06:35:23] + decoder: hevc 8-bit hwaccel (nv12, videotoolbox_vld)

when encoding.

Right - when opening an AV1 file, I don't get the scan: supported video decoders: avcodec hwaccel message - but I do get it for HEVC input, and with HEVC input it's clearly visible that it uses videotoolbox in the log.

So my main guess is, as before, the fact that the usually default decoder (libdav1d) doesn't have the hwaccel callbacks/hooks, but to do hwaccel decoding, it requires switching to a different avcodec decoder. Within ffmpeg the command line tool, this switch happens automatically, but it's probably dependent on the actual API usage patterns.

galad87 · 2024-10-09T06:01:52Z

Whoops, right, decoders 😆

Nomis101 · 2024-10-10T20:47:53Z

Thanks @mstorsjo, for testing this. Not entirely sure why it did not work for you, but thanks for your detailed explanations. Sadly I have no way to debug this at the moment by myself.

oscarbg · 2024-11-10T05:07:49Z

wanted to test on M4 but build artifact:
https://github.com/HandBrake/HandBrake/actions/runs/11204107074?pr=6339
is already expired..
can make a rebuild of this PR so new artifact is produced to test?
thanks..

oscarbg · 2024-11-11T00:44:20Z

thanks!

Nomis101 · 2024-11-12T20:28:03Z

Right - when opening an AV1 file, I don't get the scan: supported video decoders: avcodec hwaccel message - but I do get it for HEVC input, and with HEVC input it's clearly visible that it uses videotoolbox in the log.

I now see the same with my new M4 Mac. It works with HEVC, but not with AV1.

So my main guess is, as before, the fact that the usually default decoder (libdav1d) doesn't have the hwaccel callbacks/hooks, but to do hwaccel decoding, it requires switching to a different avcodec decoder.

Actually I have no idea how to do that, but maybe somebody else has.

github-actions · 2025-02-11T00:06:03Z

Hello,

Warning

This pull request appears to be inactive and will be automatically closed within 10 days if no further activity is detected.
If you wish this issue to remain open, please request the stale label to be removed and an appropriate label assigned.

Thank You,
The HandBrake Bot

Nomis101 · 2025-02-26T19:20:00Z

This seems to work in combination with the PR from #6671. I am then seeing

+ decoder: av1 10-bit hwaccel (yuv420p10le, (null))

on my M4 Mac if I open and encode an AV1 video file.

galad87 · 2025-02-26T19:24:33Z

Does it work if you use a full HW path? Without software encoders and filters? It should display something like

+ decoder: av1 10-bit hwaccel (p010le, (videotoolbox))

Nomis101 · 2025-02-26T22:04:13Z

Sadly not. Without filters, but with software encoders I get

+ decoder: av1 10-bit hwaccel (p010le, (null))

If I then also disable the software encoders I only get

+ decoder: av1 10-bit (yuv420p10le)

aXmeD2 · 2025-03-18T01:10:29Z

Hello @Nomis101,
Did I understand correctly that this patch will allow encoding video on the GPU, i.e. adds the AV1 (videotoolbox) option?

Can you please make build one more time and share instruction how to check on M3 and share results.

Nomis101 · 2025-03-18T06:57:57Z

Hello @Nomis101, Did I understand correctly that this patch will allow encoding video on the GPU, i.e. adds the AV1 (videotoolbox) option?

Can you please make build one more time and share instruction how to check on M3 and share results.

It is about decoding. So if you open an AV1 encoded video file. Not encoding. And at the moment this patch does not work correctly, as is.

aXmeD2 · 2025-03-18T07:07:53Z

It is about decoding. So if you open an AV1 encoded video file. Not encoding. And at the moment this patch does not work correctly, as is.
@Nomis101,
Oh, sorry for the misunderstanding and thank you so much for the explanation!

So at the moment there is no chance to add encoding AV1 (videotoolbox) for M3/M4?

Nomis101 · 2025-03-18T20:58:42Z

It is about decoding. So if you open an AV1 encoded video file. Not encoding. And at the moment this patch does not work correctly, as is.
@Nomis101,
Oh, sorry for the misunderstanding and thank you so much for the explanation!

So at the moment there is no chance to add encoding AV1 (videotoolbox) for M3/M4?

There is sadly no support in M3/M4 for AV1 encoding.

Nomis101 · 2025-06-13T16:58:01Z

Does it work if you use a full HW path? Without software encoders and filters? It should display something like

+ decoder: av1 10-bit hwaccel (p010le, (videotoolbox))

I've tried today again with the latest master and the two MR and now I am getting:
+ decoder: av1 10-bit videotoolbox hwaccel (yuv420p10le, (null))

galad87 · 2025-06-17T08:08:19Z

Still no full HW path, I reworded that line a bit after refactoring some QSV only code.

Nomis101 · 2025-06-17T16:05:01Z

OK. But its the same syntax I am seeing for H264 or HEVC sources:

decoder: hevc 10-bit videotoolbox hwaccel (p010le, (null))

decoder: h264 8-bit videotoolbox hwaccel (nv12, (null))

And this should definitely work, shouldn't it?

galad87 · 2025-06-17T17:45:22Z

If there isn't any cpu filter, yes. If there is (null) at the end it means it's going from gpu to cpu.

Nomis101 · 2025-06-17T18:41:30Z

How can I see if there is any cpu filter?

galad87 · 2025-06-18T05:47:39Z

Here's a list of the supported ones #5721

galad87 · 2025-06-18T15:40:21Z

By the way, if the hardware decoder works, it should just work in the full GPU path if you disable the unsupported filters and use the VideoToolbox encoders. I see no reason for it not working.

Nomis101 · 2025-06-18T16:28:24Z

By the way, if the hardware decoder works, it should just work in the full GPU path if you disable the unsupported filters and use the VideoToolbox encoders. I see no reason for it not working.

I always disable all filters in my tests. If I in addition use a VT encoder, then I get:
+ decoder: av1 10-bit videotoolbox hwaccel (p010le, videotoolbox_vld)

Support AV1 hardware acceleration on Apple M3 and newer

4d3f304

Nomis101 requested a review from galad87 October 6, 2024 16:22

Fix A17-AV1-videotoolbox.patch version mismatch

c015bec

Merge branch 'HandBrake:master' into AV1-M3

c544442

Merge branch 'HandBrake:master' into AV1-M3

d9d782d

github-actions bot added the Stale label Feb 11, 2025

galad87 removed the Stale label Feb 11, 2025

dashsantosh-mcw mentioned this pull request Feb 26, 2025

hwaccel: enable av1 and vp9 decoder support #6671

Merged

3 tasks

Nomis101 added 2 commits February 26, 2025 20:46

Merge branch 'HandBrake:master' into AV1-M3

92142e0

Renumber patch

42b017d

galad87 closed this in #6671 Jul 4, 2025

galad87 reopened this Jul 4, 2025

galad87 merged commit 61a2aec into HandBrake:master Jul 4, 2025
6 of 11 checks passed

Nomis101 deleted the AV1-M3 branch July 8, 2025 20:48

Support AV1 hardware acceleration on Apple M3 and newer #6339

Support AV1 hardware acceleration on Apple M3 and newer #6339

Uh oh!

Conversation

Nomis101 commented Oct 6, 2024

Uh oh!

galad87 commented Oct 8, 2024

Uh oh!

bradleysepos commented Oct 8, 2024

Uh oh!

galad87 commented Oct 8, 2024

Uh oh!

bradleysepos commented Oct 8, 2024

Uh oh!

bradleysepos commented Oct 8, 2024

Uh oh!

mstorsjo commented Oct 8, 2024

Uh oh!

bradleysepos commented Oct 8, 2024

Uh oh!

mstorsjo commented Oct 8, 2024

Uh oh!

galad87 commented Oct 9, 2024

Uh oh!

mstorsjo commented Oct 9, 2024

Uh oh!

galad87 commented Oct 9, 2024

Uh oh!

Nomis101 commented Oct 10, 2024

Uh oh!

oscarbg commented Nov 10, 2024

Uh oh!

oscarbg commented Nov 11, 2024

Uh oh!

Nomis101 commented Nov 12, 2024

Uh oh!

github-actions bot commented Feb 11, 2025

Uh oh!

Nomis101 commented Feb 26, 2025

Uh oh!

galad87 commented Feb 26, 2025

Uh oh!

Nomis101 commented Feb 26, 2025

Uh oh!

aXmeD2 commented Mar 18, 2025

Uh oh!

Nomis101 commented Mar 18, 2025

Uh oh!

aXmeD2 commented Mar 18, 2025

Uh oh!

Nomis101 commented Mar 18, 2025

Uh oh!

Nomis101 commented Jun 13, 2025

Uh oh!

galad87 commented Jun 17, 2025

Uh oh!

Nomis101 commented Jun 17, 2025

Uh oh!

galad87 commented Jun 17, 2025

Uh oh!

Nomis101 commented Jun 17, 2025

Uh oh!

galad87 commented Jun 18, 2025

Uh oh!

galad87 commented Jun 18, 2025

Uh oh!

Nomis101 commented Jun 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants