I have struggled with this all year while streaming in 4k 60p. All Ive read is that this is a bandwidth issue so that the problem never gets solved....well its not. I have installed network down detector and had it launched while streaming. Zero network connection problems when Real Time Buffer issue hits. I have IMMEDIATELY done a speed test while stream is blinking orange...800mbps upload while Im only sending 40mbps. Ive even done an NDI stream on the same network in 1080p. While streaming the 4k 60p 40mbps I simultaneously ndi it to another laptop on the same switch and stream in 1080p 12mbps. When the 4k blinks orange the 1080p is having ZERO ISSUES. When the 4k 60p 40mbps starts blinking orange it can take between 15 seconds to 2 minutes to fix itself....HOWEVER when I immediately click stream to end the stream the moment it turns orange and then immediately click stream again to restart it.....it works immediately. It does not blink orange for a single moment when I stop and restart it in the middle of blinking. The internet does not magically fix itself the moment I end and restart the stream...every single time. This has got to be some kind of a Vmix issue.

Im streaming with an amd 3600, nvidia 5700 ti, 32 GB ram, Elgato 4k60 PRO v2. HEVC, P1 fastest. GPU never gets above 50% usage and CPU never gets above 60%.

Can anyone PLEASE help me fix this.

You compare apple with pears.
Local network with speed test vs rtmp streaming.
If you want to check something on the path to the streaming server I suggest you have an MTR running in parallel. It could show you potential bottlenecks on the way to the server. You have to ping the server to get it's IP.
Second, there could be some antivirus/firewall/IPS software either on the computer or in the network that can't deal with streams. Streams can pose a problem for AV as the software waits for the end of the file which is never coming and thus causes a buffer overflow. Stopping and restarting the stream gives you a zero buffer again until it's full again causing the stream to stall again. If the stall happens always at xx minutes after start you know that some buffer runs full reproducible. If reduce the bandwidth to half and the xx amount of minutes double you have another hint that there is something in between you and the server.
Third option could be the server itself.

I hope this helps to get closer to the issue.