Can we customize BAR 1 size via linux kernel/module params?

[Qubitium]

Right now I am testing A100 80GB PCIE on a consumer board and it is allocating 128GB BAR 1 size. Unfortunately this is causing pcie issues where other gpu devices are unable to allocate BAR due to the A100 hogging all the available space.

I have searched everywhere but unable to find a solution to have the A100 allocate only 16GB of BAR 1 for example. Via lspci output, the A100 supports 16GB and lower BAR 1 size via power of 2 increments.

Is a way to lower the A100 or any other Nvidia GPU to a lower but supported BAR 1 size?

Thank you for any assistance.

[aaronp24]

You might be able to use the NVreg_EnableResizableBar feature but you'll need to modify the code if you want to specifically request a smaller size: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/nvidia/nv-pci.c#L170

That code always tries the largest size first and goes down from there if it doesn't fit. You'll need to disable some of the bits in the sizes variable in order to resize it smaller. (Note: I haven't tried this)

[Qubitium]

@aaronp24 This worked perfectly. Unfortunately, it did not fix my core issue. Very strange. Maybe I can pick you brain for a bit.

Before

1. A100 GPU assigned 128GB BAR 1 (it is at max 128GB bar limit)
2. 2X4090 GPU on same pcie root, assigned 256MB BAR 1 (it is capable of 32GB bar via lspci)

So I though decreasing the A100 bar size at kernel/driver load will solve this, using your tip.

After compiling custom nvidia driver and making sure A100 only assigned 32GB.

1. A100, assigned 32GB BAR1 (success!)
2. 2x4090, on same pcie root, still assigned 256MB (?)

So then I tried using linux kernel (6.6.5) sysfs method to resize bar of the 4090 to 1GB. I figured, I just freed 90GB of bar/mem space from pcie table there should be enough for tiny 1GB for the 2x4090 right? Wrong, I still get "no space left on device" when attempting to modify bar1 size via sysfs.

Pcie/bar seems like voodoo magic to me. Any possible reason for the rejection of 2x 1GB bar1 for the 4090s after 90GB freed for A100?

[Qubitium]

After reducing A100 bar1 to 32GB, I also tried using sysfs to remove the pcie 4090 devices and pcie rescan to add them back to retrigger bar1 increase, still same 256MB limit.

[Qubitium]

I can freely use sysfs to runtime resize A100 bar1 within all the supported ranges. Yet 4090 only allows 256MB and fails with no space left on device for all other supported sizes.

[aaronp24]

What does pci_rebar_get_possible_sizes report for your 4090s? ReBAR could be disabled in the VBIOS, although I thought it was enabled for all 40 series cards.

Oh, it might also be worth checking your PCIe topology. I think if both 4090s are on the same PCIe bridge, it might not be able to move the BARs if it can't also move the memory window of the upstream bridges due to the other GPU still being present. I'm not sure how the upstream kernel handles that kind of situation.

[atedjarati]

@Qubitium I am running into similar issues on my PC. I am running aarch64 and need to reduce the BAR 1 size on my graphics card. Can you give me your instructions on how you used sysfs to do this? Thank you!!

[Qubitium]

@web3creator Why would you want to do this? Are you able to pcb hack 4090 to 64GB?

4090 has 24GB max. BAR1 size appears to be power of 2 so it's 16GB, 32GB (24GB 4090) and 64, 128, etc. Having BAR1 that exceeds physical vram doesn't help imho.

[web3creator]

@Qubitium 64GB is very helpful for optimizing my program. I can't find any tool to set the size of bar1 to 64GB. The current maximum is 32GB.

[Qubitium]

@web3creator See my above and use 14 for 64GB. You may need to hack the driver code and up the max ranges. It may only give you the power of 2 closes to your gpu vram. So 24GB gpu is max 13.

# do resize of BAR1
echo 14> $resource

[web3creator]

@Qubitium

I execute this and get nothing

[web3creator]

@web3creator See my above and use 14 for 64GB. You may need to hack the driver code and up the max ranges. It may only give you the power of 2 closes to your gpu vram. So 24GB gpu is max 13.

# do resize of BAR1
echo 14> $resource

hack the driver I don't know how to do it