As for now, vMix by default is not able to use more than 2 cameras for 4k60 replay. So better stick to FullHD along your signal chain for 4-5 cameras in instant replay.

Also, Ryzen isn't really the best choice here in terms of pcie lanes and memory channels. You'd be better with Intel X299 and X-series Intel-Core processors. They have more pcie lanes and also more memory channels.
If you have an 30-series NVIDIA graphics-card (16 pci lanes) and 2x Decklink 8k Pro (2x 8 lanes) you are already well above the total pci-lane count of your ryzen.
So the bottleneck really would be your mobo only using a reduced amount of pcie-lanes for each card or managing all the data, prioritizing and communicating in serial. Both would be more or less suboptimal for real-time use.

GPU and RAM are well above what you need, vMix recommends a GTX 1080 ti or higher and at least 16GB DDR4 for 8 channel HD- or 2 channel 4k replay.

In addition, temperature could also be a problem, since the Decklink cards only have a maximum operating temperature of 50 °C. If you have a 3080 ti next to them blasting hot air close, that maximum temperature will be reached easily. A blower-style graphics-card would be a better in this case.

I hope I could provide some helpful information for you
Jan