iGPU Passthrough

SUSE Virtualization
1

Context: I am trying to passthrough my iGPU (radeon 680m) to harvester VMs.

After some trials I have managed to make harvester passthrough the GPU. On the host side GPU gets binded to vfio-pci, VM boots and I can see the GPU in guest VMs. To do so I had to manually edit harvester kernel parameters blacklisting amdgpu driver for vfio-pci to correctly bind (https://docs.harvesterhci.io/v1.4/troubleshooting/os/). Otherwise, whenever I try to enable passthrough or manually unbind amdgpu, my harvester node crashes (as expected, as the device is both host-owned and in use).

Now I am facing another issue, where the GPU cannot get initialized in the guest VM due to weird errors accessing the BIOS (BAR6). That also should be a “known” problem with consumer grade GPUs (no vGPU support). My guess is that disabling amdgpu drivers in the grub prevents the GPU to be loaded at all in when host starts up, thus requiring the vBIOS to be injected into the VM (or maybe, it would not be passed to kubevirt/qemu and would require manual injection anyway).

So far, I have managed to get the rom file for my iGPU (link below) and I have it mounted as a configmap into my VM (such that file is visible in virt-launcher containers and can be passed as input to qemu). Now I am trying to edit the xml spec in kubevirt pod to get it loaded. I think kubevirt sidecars is the only approach I have and I need something very similar to https://github.com/kubevirt/kubevirt/issues/11552.

Anyone went that down this rabbit hole who can point me in the right direction?

For referece, I am trying to follow this: https://github.com/isc30/ryzen-7000-series-proxmox

Additional notes:

  • I have amd_iommu on and other iommu parameters enabled (default in harvester)
  • GPU is isolated in its own IMMOU group
  • Combinations of different parameters to disable framebuffer (vesafb:off, efifb:off, initsys fb off) did nt help
2

I figured out, vBIOS was indeed the issue for the GPU not being initialized in the guest VM. Kubevirt sidecar was the rigth approach, had to upgrade to harvester 1.4.0 (bumping kubevirt to v1.2.0) to make sidecar-shim work.

3

Posting the solution here in case just in case.

I added this annotation to my VM to use sidecar-shim

hooks.kubevirt.io/hookSidecars: >-
       [{
          "image": "quay.io/kubevirt/sidecar-shim:v1.2.0-rc.0",
          "args": ["--version", "v1alpha2"],
          "configMap": {"name": "gpu-vbios-xml-add", "key": "gpu_vbios.sh", "hookPath": "/usr/bin/onDefineDomain"}
       }]

This allows to attach a sidecar to Kubevirt VM and mount a configmap, from which it will execute a python/go/bash script. In my case it’s shell script updating the libvirt XML spec (similar to the shmem solution).

My configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-vbios-xml-add
  namespace: default
data:
  gpu_vbios.sh: |
    #!/bin/sh
    tempFile=`mktemp --dry-run`
    echo $4 > $tempFile
    sed -i 's|<alias name="ua-hostdevice-harvester-0000e4000"></alias>|<alias name="ua-hostdevice-harvester-0000e4000"></alias><rom bar="on" file="/var/run/kubevirt-private/config-map/gpu-vbios/vbios_6900HX.bin"></rom>|' $tempFile
    sed -i 's|<alias name="ua-hostdevice-harvester-0000e4001"></alias>|<alias name="ua-hostdevice-harvester-0000e4001"></alias><rom bar="on" file="/var/run/kubevirt-private/config-map/gpu-vbios/AMDGopDriver_6900HX.rom"></rom>|' $tempFile
    cat $tempFile