# GPU passthrough in Harvester
**URL:** https://forums.suse.com/t/gpu-passthrough-in-harvester/43274
**Category:** SUSE Virtualization
**Created:** [May 28, 2024, 2:29am UTC](https://forums.suse.com/t/gpu-passthrough-in-harvester/43274 "2024-05-28T02:29:43Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Slavik\_Fursov](https://forums.suse.com/u/Slavik_Fursov)
#### Post date: [May 28, 2024, 2:29am UTC](https://forums.suse.com/t/gpu-passthrough-in-harvester/43274/1 "2024-05-28T02:29:43Z")
Does anyone have experience passing GPU to VM in Harvester?
I know that Harvester supports vGPU, but only few graphic card models support vGPU.
I have nVidia Quadro P5000 (Pascal) 16GB RAM and it doesn’t support vGPU.
I know, there are [hacks](https://gist.github.com/kralicky/0f9994526eac7ddc1808bcbfea6a8444) and [workaround](https://gitlab.com/polloloco/vgpu-proxmox). I tried few things, but was unable to figure out everything.
Here is my experience passing nVidia Quadro P5000:
**Section 1** (not sure if needed):
- SSH to Harvester with GPU
```auto
sudo su
# make sure the card is present on the host:
lspci | grep -i nvidia
0000:d5:00.0 VGA compatible controller: NVIDIA Corporation GP104GL [Quadro P5000] (rev a1)
0000:d5:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
# prevent host from using GPU card:
# https://forums.unraid.net/topic/99478-solved-gpu-passthrough-issue-bar-0-cant-reserve/?do=findComment&comment=923145
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
```
**Section 2**
in Harvester web UI:
- Advanced → Addons → picdevices-controller → Enable
- wait while all PCI devices are listed
- enable passthrough for nvidia devices
- create VM. I used Ubuntu 22 cloud image with 4 cores and 12GB RAM, no GUI, server image
- add PCI devices. I only added “VGA compatible controller: NVIDIA Corporation GP104GL [Quadro P5000]”. I didin’t add “Audio device: NVIDIA Corporation GP104 High Definition Audio Controller”
- start the VM:
**Section 3**
SSH to VM and run:
```auto
# confirm the GPU is present in VM:
lspci | grep -i nvidia
0a:00.0 VGA compatible controller: NVIDIA Corporation GP104GL [Quadro P5000] (rev a1)
```
Download nVidia drivers by following instruction here:
> **[Linux x64 (AMD64/EM64T) Display Driver | 550.78 | Linux 64-bit | NVIDIA](https://www.nvidia.com/Download/driverResults.aspx/224350/en-us/)**
>
> Download the English (US) Linux x64 (AMD64/EM64T) Display Driver for Linux 64-bit systems. Released 2024.4.25
```auto
# see the status:
nvidia-smi
```
I also installed CUDA toolkit from here:
> **[CUDA Installation Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#ubuntu)**
>
> The installation instructions for the CUDA Toolkit on Linux.
Currently running OpenAI Whisper. Works great.
Next I want to install Docker on that Ubuntu host and run Frigate and Immich.
I would be curious to hear your experience with supported vGPU on Harvester. I’m currently considering getting system with nVidia RTX 5000.
Or may be someone figured out how to run vGPU even on non-supported cards?
Another question: since my P5000 doesn’t support vGPU, should I try to install Ubuntu on that server bare-metal, install regular nVidia drivers and join Harvester cluster as a worker node with GPU support? Or that will not work?
---
### Author:  [@Slavik\_Fursov](https://forums.suse.com/u/Slavik_Fursov)
#### Post date: [June 12, 2024, 2:55am UTC](https://forums.suse.com/t/gpu-passthrough-in-harvester/43274/2 "2024-06-12T02:55:56Z")
After the node was rebooted, the nVidia PCI pass-through stopped working.
That’s because setting from Section 1 was gone.
In the kernel log I saw this:
> dmesg -T
> [Wed Jun 12 02:42:37 2024] vfio-pci 0000:b3:00.0: vfio\_ecap\_init: hiding ecap 0x19@0x900
> [Wed Jun 12 02:42:37 2024] vfio-pci 0000:b3:00.0: BAR 1: can’t reserve [mem 0xe0000000-0xefffffff 64bit pref]
> [Wed Jun 12 02:43:45 2024] vfio-pci 0000:b3:00.0: BAR 1: can’t reserve [mem 0xe0000000-0xefffffff 64bit pref]
> …
So, it looks like Section 1 needs to be executed after every reboot. I need to figure out how to add these steps, so they’re executed automatically.
---
### Author:  [@Felipe\_De\_Bene](https://forums.suse.com/u/Felipe_De_Bene)
#### Post date: [November 20, 2024, 3:08am UTC](https://forums.suse.com/t/gpu-passthrough-in-harvester/43274/3 "2024-11-20T03:08:20Z")
@Slavik_Fursov have you figured out how to do this? your comments help me a lot.
---
### Author:  [@Felipe\_De\_Bene](https://forums.suse.com/u/Felipe_De_Bene)
#### Post date: [November 20, 2024, 5:01am UTC](https://forums.suse.com/t/gpu-passthrough-in-harvester/43274/4 "2024-11-20T05:01:36Z")
I mean in the persistence part specifically. Also I am curious if there’s a way to export some gpu metrics to prometheus. Have you exploded any of that?
---
### Author:  [@Slavik\_Fursov](https://forums.suse.com/u/Slavik_Fursov)
#### Post date: [November 30, 2024, 5:47pm UTC](https://forums.suse.com/t/gpu-passthrough-in-harvester/43274/5 "2024-11-30T17:47:15Z")
One reason why I have not done persistence is to prevent myself from locking out.
Think about it:
- you enable persistence
- host started to have some issues / need to fix things via terminal
- you attach monitor, keyboard … and you can’t see anything because GPU is disconnected from host OS. Reboot will not help, because persistence enabled.
Anyway, I think to enable persistence you need to apply cloud-init config like this (modify hostname):
For nVidia Quadro P5000:
```yaml
# kubectl apply -f VM-P5000/release-p5000.yaml
apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
name: release-gpu
spec:
matchSelector:
kubernetes.io/hostname: "t7920"
filename: 99_p5000.yaml
contents: |
stages:
network:
- name: "disconnect GPU from host OS"
commands:
- echo "disconnecting GPU from host OS" > /dev/kmsg
- echo 0 > /sys/class/vtconsole/vtcon0/bind
- echo 0 > /sys/class/vtconsole/vtcon1/bind
- echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
```
For nVidia RTX 3090:
```yaml
apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
name: release-gpu
spec:
matchSelector:
kubernetes.io/hostname: "t5820"
filename: 99_rtx3090.yaml
contents: |
stages:
network:
- name: "disconnect GPU from host OS"
commands:
- echo "disconnecting GPU from host OS" > /dev/kmsg
- echo 0 > /sys/class/vtconsole/vtcon0/bind
```
I have not tested that. So, let me know here, how that works…