So we build a brand new Machine for Livestreaming a Visual radio show twice a week. ( our i7 machine was not pulling trough anymore... It just needed more power )
Vmix 4K license ( not activated the license yet due too we might need to swap hardware or even look for a different machine ) My i7 has Vmix Pro license installed
Gigabyte X399 Aorus Pro
AMD Ryzen Threadripper 2920X ( 12 Core 3.5Ghz )
16Gb 3200-16 Corsair memory
Gigabyte RTX 2070 Windforce 8gb Videocard
NVMe 1TB PCIe M.2 SSD ( 100Gb for Windows, 800Gb for video playback)
2x BMD Decklink 8K Pro ( 1 for ingest 4 channels, 1 for output 2 channels ) ( Running latest FW/Desktop video version )
850W Power supply Corsair
Win10 Pro
Running 1080P25 at the moment as everything is for streaming to Youtube and other CDN's. We use Restream so 1 output.
FPS is constant 25 Fps. Render time most of the time 7 till 13 ms.. But spikes sometimes up to 50/60 ms for no known reason. Gpu Cahche Mem depending on how many sources but up to 80%
I optimized GPU settings like described on the Vmix. Nvidia Studio Driver 442.92 Nvidia Experience is not installed, Power settings to maximum.
We ran the Vmix Diagnostic Tool and it came not up with anything special.. Only this>> UMPDC.dll: (C:\WINDOWS\System32\UMPDC.dll)
So basically we have 2 SDI inputs in use all the time. ( External SDI from an Atem in the Studio ) and a SDI from a Computer running Zoom. SDI 3 / 4 Are spares if we need to put more sources on it for future use.
Then we have a PGM output to a local monitor and a Hyperdeck recorder for external recording of individual sets
And External 2 is doing SDI output for the screen behind the Co-host
NDI output for the Clock that offers current time, Video left time and a preview of the PGM signal. ( Ingested as Source from Vmix PGM out NDI )
Audio main bus For program out
Audio A bus only for pre-talkback to vcall to check with the new caller we have audio from them.
Audio B bus is everything back to radio studio without there talk. ( bus minus )
Sources:
We got a 10 second animated overlay that loops the whole show. ( Drops a frame every start on the loop. So every 10 seconds adds a dropped Source frame )
We use 3 MIX units as source. Mainly to control the Pip window sources ( from a streamdeck ) and Mix4 is used as a Seamless router to route sources to the External2 output (Screen behind CoHost)
Have a stinger in place, Some Text media for the clock, some static images,
Got an Animated PIP.mp4 which is 30mb that is also running in loop all the time. As does the animated Overlay.
And then we have about 20 videos we have to play in a 11 hour shift. Most of the time MP4 encoded in 6/10 mbit codecs.
So about the show yesterday we ran about 3/4 hours without any output render drops. Only thing that did drop was the Animated bug.. But it does constant every 10 seconds a Source drop.. No worries there.. Render times also kept far below 20ms..
But after the 3rd Vcall it started again dropping frames on the output. And i could not solve it anymore.. and we saw unexplained Render time spikes that could run up to 50/60ms..
End of the evening we had about 814 render dropped on the outputs, 509 on the main Studio SDI input...
And a lot of frame dropped on audio.. This is what is the biggest issue for us.. A video frame that drops will not be noticed that much.. But the audio sample that is gone will be heard directly while playing audio sets..
We now did 3 shows in a row and always the errors seem to start a few hours into use. Also often linked to Vcall's or problems start when using / after the vcall..
I have some feeling some buffer is filled and not cleaned / released anymore.. Resulting in buffer overflows..
We are closing media that we do not use anymore.. So all videos that are done are removed to clear out more Cache Memory for the GPU..
At the end of the evening i deleted all "inputs" to the point i had only a few basic things running and 1 MP4 video.. And still kept dropping about 3/10 frames a minute..
We were thinking that it might / would have been a heat issue.. but we put so much fans in that machine that it is running cool.
Also looking at Windows Task manager the CPU/GPU/Memory load is not an issue.. Running about 20% CPU / 25/30% GPU 9Gb Memory
Rebooting the machine is no option as it is our streaming outpoint. But i really wondering what is going on here..
I ran the LatencyMon and came out in 40 minutes while running Vmix show with 360uS Current int latency, 971uS Highest measure latency, 456uS Highest ISR exc, 971uS Highest DPC Exc, Hard pagefault count 10754.. Still all green and OK what the software says.
GPU is in the first 16x slot. Next to it is a 8x slot running a decklink 8k. Next to that is a 16x slot for a second GPU which is not populated, followed by the 4rd slot also 8x slot for a Decklink 8k. This is how the manufacturer tells it on the website that it has 2 Full 16x and 2 Full 8x slots. ( the reason we choose Threadripper and this Motherboard )
Putting down external outputs, NDI, or even streaming etc does not really impact dropped frames we experience once it is dropping.. It stays dropping randomly..
1- So any advice on how to proceed in debug?
2- Can we monitor other caches or buffers?
3- Could we render videos in a different format that is easier to handle on the machine then MP4?
4- Is there a Bug in Vcall that fills up a buffer or is responsible for some kind of Buffer overflow?
5- What do the 4(5) values mean in the statistics window under Video.. Are these render times for the last 4 frames or what am i looking at?
6- Why is media that is not used still holding 4 frames in the Que?
196.png
(1,530kb) downloaded 32 time(s). Image 17.png
(1,580kb) downloaded 24 time(s).