I have a friend who uses multiple displays on his 3080 using the Nvidia driver. They work for him.

It sounds like your issue is related to the dma buf issue that the kernel developers created. I thought that was fixed in recent versions of the open source Nvidia driver. What was the last driver you tried?