OVMF PCIE passthrough with large VRAM GPU

Recently I got a GPU Tesla M40 24GB, and I bought this card because I saw someone on YouTube said he can get this card gaming. Although he used another iGPU to pass this GPU’s output to the screen, but since I wasn’t planning to put any of the compute power in my house, I put them in a server room in my town, so I wasn’t going to use a physical monitor anyway. Like any other computer geek, I’m going to virtualize my server, and pass this GPU into a VM, then play my game in that machine.

If everything is only that simple.

You see, for every “GPU passthrough” topic you searched in Google, almost everyone told you to enable “CPU Virtualization” in your server, load vfio kernel module, blacklist GPU vendor graphics driver, change some system setting so your GPU uses vfio driver instead of no driver, then pass the card into VM, then all is well.

Yes this method works, for lots of graphics cards. But this method also don’t tell you, that it only works if you don’t have a GPU with large VRAM, or you’re trying to pass multiple GPU into your VM. As my card got 24G VRAM with it, the method above doesn’t work for me.

After I tried to pass GPU into VM, I got this error message in Windows.

With Google’s help, I found out that this Code 12 issue may be related to IO.

Since I got no Windows debug experience, I decided to switch to Ubuntu Linux, see if I can find some more information there.

root@ubuntu:~# dmesg | grep nouveau
[    2.373509] nouveau 0000:01:00.0: enabling device (0000 -> 0002)
[    2.378767] nouveau 0000:01:00.0: NVIDIA GM200 (1203d0a1)
[    2.496491] nouveau 0000:01:00.0: bios: version 84.00.56.00.03
[    2.499154] nouveau 0000:01:00.0: pmu: firmware unavailable
[    2.515956] nouveau 0000:01:00.0: fb: 24576 MiB GDDR5
[    2.516028] nouveau 0000:01:00.0: bus: MMIO read of 00000000 FAULT at 6013d4 [ IBUS ]
[    2.543276] nouveau 0000:01:00.0: bar: one-time init failed, -12
[    2.543599] nouveau 0000:01:00.0: init failed with -12
[    2.543646] nouveau: DRM-master:00000000:00000080: init failed with -12
[    2.543832] nouveau 0000:01:00.0: DRM-master: Device allocation failed: -12
[    2.547829] nouveau: probe of 0000:01:00.0 failed with error -12

As we can see in dmesg, the system can read the card correctly, but when OS tried to initialize the card, it failed, with error message related to MMIO.

This issue can be traced to MMIO with QEMU, and there are already some discussion in the world. After some more digging, I found out that in order to make GPU passthrough work, I need to enable Above 4G decoding BIOS function in VM. Setting this option in bare metal is quite easy, as modern BIOS include this option in their menu. But when in QEMU, OVMF BIOS doesn’t include this option in VM, and we would need to enable it from the hypervisor.

I found out how, you need to add this parameter in your qemu-kvm parameter. You can refer to this topic for more information

-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536

And now you should be able to pass GPU with large VRAM into your VM, or pass multiple GPU into your system. Happy gaming :)