Skip to content

Instantly share code, notes, and snippets.

@paul-vd

paul-vd/readme.md

Last active December 29, 2024 04:04
Show Gist options
  • Save paul-vd/5328d8eb2c626dff36ee143da2e85179 to your computer and use it in GitHub Desktop.
Save paul-vd/5328d8eb2c626dff36ee143da2e85179 to your computer and use it in GitHub Desktop.
Download ZIP
GPU Passthrough Fedora
Raw
readme.md

Simular Guides

Usefull Commands

  • lspci -nnk list pci devices
  • sudo grub2-mkconfig -o /etc/grub2-efi.cfg updates the grub
  • sudo call-a-friend @ballas calls the best knowledge base, fuck wiki 🤣

Detach the GPU from the host

Retrieve the GPU id's

First you will have to find the GPU id's that you would like to passthrough, you can do this by running in the terminal:

# For AMD GPU's:
$ lspci -nnk | grep Navi

# For nvidia GPU's:
$ lspci -nnk | grep NVIDIA

Example output, here we are interested in the VGA compatible controller (1002:73bf) and the Audio device (1002:ab28)

0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]

Blacklist id's and enable iommu

First you will have to modify the default grub file.

$ sudo gedit /etc/default/grub

We want to modify the GRUB_CMDLINE_LINUX

# for amd  gpu replace GPU_TYPE with `amdgpu`.
# for nvidia gpu replace GPU_TYPE with `nouveau`.
# if you have and intel replace `amd_iommu` with `intel_iommu`.
# `vfio-pci.ids` are equivalent to the id's found in the previous step.

# values to add
`amd_iommu=on`
`video=efifb:off` // disable gpu on boot
`kvm.ignore_msrs=1`
`rd.driver.pre=vfio-pci` // this preloads the vfio-pci

- GRUB_CMDLINE_LINUX="rhgb quiet"
+ GRUB_CMDLINE_LINUX="rhgb quiet rd.driver.blacklist=GPU_TYPE modprobe.blacklist=GPU_TYPE video=efifb:off amd_iommu=on rd.driver.pre=vfio-pci kvm.ignore_msrs=1 vfio-pci.ids=1002:73bf,1002:ab28"

Then we need to update grup

sudo grub2-mkconfig -o /etc/grub2-efi.cfg

We then need to edit the dracut config

$ sudo gedit /etc/dracut.conf.d/local.conf

add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd "

$ sudo dracut -f --kver $(uname -r)

Rebooot!

Attach the GPU from the guest

Install Virtual Machine

$ sudo dnf groupinstall virtualization

Then you reboot 🦸

  1. Open up Virtual Machine Manager
  2. Create new virtual machine
  3. Local install media (ISO image or CDROM) -> Select the ISO to windows10 or windows11
  4. Assign Ram & CPU
  5. Tick the Customize configuration before install

If you are running windows 11 you will need a secure boot and tpm module.

  1. Under overview add the following config
Chipset: q35
Firmware: x86_64:/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd
  1. Under CPUs, untick the copy host cpu configuration then for a example use the following config for a Ryzen 5900x
Model: host-passthrough
Topology
[X] Manually set CPU topology
Sockets: 1
Cores: 6
Threads: 2
  1. Add a TPM module by clicking on Add Hardware > TPM
Model: TIS
Backend: Emulated device
Version: 2.0
  1. Then passthrough the GPU and if needed the GPU Audio under the Add Hardware > PCI HOST Device image

And then you can begin installation. Bobs your uncle 🤠

@ciansen
Copy link

ciansen commented Dec 9, 2022

Hey, simple question, I have like two GPUs (one integrated from cpu):

00:02.0 Display controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 05)
DeviceName: Onboard - Video
Subsystem: ASUSTeK Computer Inc. Device 8694
Kernel driver in use: i915

01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. Device 1612
Kernel driver in use: nouveau
Kernel modules: nouveau

How does it work? Will my computer use the integrated GPU from the CPU if I disable the GPU (main, rtx 3080) from Booting?
Best regards
cian

@kaem-e
Copy link

kaem-e commented Feb 17, 2023

Thank you so much for making this @paul-vd !

@kaem-e
Copy link

kaem-e commented Feb 17, 2023

Hey, simple question, I have like two GPUs (one integrated from cpu):

00:02.0 Display controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 05)

DeviceName: Onboard - Video
Subsystem: ASUSTeK Computer Inc. Device 8694
Kernel driver in use: i915

01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080] (rev a1) Subsystem: ZOTAC International (MCO) Ltd. Device 1612 Kernel driver in use: nouveau Kernel modules: nouveau

How does it work? Will my computer use the integrated GPU from the CPU if I disable the GPU (main, rtx 3080) from Booting? Best regards cian

Yup, you will have to plug in display outputs to both your motherboards display out and your gpu's output. Also make sure that the gpu you're passing through exists in its own iommu group,

Essentially just run this script:

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

and see if the output puts your gpu you want to pass through in its own iommu group entirely (Should look something like this)

IOMMU Group 16:
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GT 640] [10de:0fc1] (rev a1)
06:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
IOMMU Group 17:

(heres a snippet from the archwiki that goes through this in detail)

@kaem-e
Copy link

kaem-e commented Apr 28, 2023

For anyone getting dracut-install: Failed to find module 'vfio_virqfd' error after the dracut config bit:

sudo gedit /etc/dracut.conf.d/local.conf

add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd "

sudo dracut -f --kver $(uname -r)

Post Linux-kernel 6.2, vfio_virqfd has been consolidated with the vfio driver so you don't need to append it in the dracut config anymore. The fixed snippet would be:

sudo gedit /etc/dracut.conf.d/local.conf

add_drivers+=" vfio vfio_iommu_type1 vfio_pci "

sudo dracut -f --kver $(uname -r)

- add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd "
+ add_drivers+=" vfio vfio_iommu_type1 vfio_pci "

@Docmine17
Copy link

Docmine17 commented Oct 4, 2023 •
edited
Loading

To block Nvidia's proprietary driver modules use:
rd.driver.blacklist=nvidia,nvidiafb,nvidia-gpu modprobe.blacklist=nvidia,nvidiafb,nvidia-gpu

Source:
https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-passthrough-to-vm/#proxmox-configuration-for-igpu-full-passthrough

Copy link

ghost commented Nov 18, 2023

Hi! I followed this guide on my laptop with Ryzen 7 4700U and its GPU but, when I start the VM the screen begins black and the computer stucked. If I remove the PCI device all works correctly. How can I fix it? Is there anything I can share to make the problem clearer?

@aut-mn
Copy link

aut-mn commented May 31, 2024

Hey, I'm trying to pass through an rx 580 of mine but when I try to run the virtual machine, I get this error:

system-x86_64: -device {"driver":"vfio-pci","host":"0000:04:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:04:00.0: group 15 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

I've checked to make sure the drivers in use are vfio-pci and are in group 15 (which they are):

IOMMU Group 15:
	[...]
	04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
	04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
	[...]

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:22fc]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:aaf0]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Anyone know of any fixes?

@kaem-e
Copy link

kaem-e commented May 31, 2024

Hey, I'm trying to pass through an rx 580 of mine but when I try to run the virtual machine, I get this error:

system-x86_64: -device {"driver":"vfio-pci","host":"0000:04:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:04:00.0: group 15 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

I've checked to make sure the drivers in use are vfio-pci and are in group 15 (which they are):

IOMMU Group 15:
	[...]
	04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
	04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
	[...]

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:22fc]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:aaf0]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Anyone know of any fixes?

@aut-mn have you included the 04.00.1 hdmi audio in the vm's passthrough settings, when passing through pcie devices you need to pass through everything that shows up in a specified IOMMU group (15 in your case)

@Myrdincx
Copy link

Myrdincx commented Sep 2, 2024

I've tried to follow this guide, but whenever I'm trying to boot I'm getting the following thing:
VFIO - User Level meta-driver version 0.3 and it's refraining me from actually booting.
Does anyone know how to fix this issue?

@Danthebee1
Copy link

When I was trying to fix the " dracut-install: Failed to find module 'vfio_virqfd' " error I edited the config file and got this error when running the "sudo dracut -f --kver $(uname -r)" command I got this error:

Screenshot_20241122_193232

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment