logo

Live Production Software Forums


Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
VisionCloud  
#1 Posted : Wednesday, September 21, 2016 6:38:58 PM(UTC)
VisionCloud

Rank: Newbie

Groups: Registered
Joined: 9/21/2016(UTC)
Posts: 2
Location: Utah, USA

Was thanked: 1 time(s) in 1 post(s)
Hey everyone, first time coming to the VMIX forums, hoping I can get some help with a problem I am having.

First off, here is the setup:

Computer:
MoBo: Asus x99-E WS USB 3.1 (X99 Deluxe II)
CPU: Intel i7 6900K
GPU: Asus GTX 1070 Strix
Ram: Corsair 128GB Dominator Platinum
Capture Card: Aja Kona 4
Capture Card2: Blackmagic Decklink Duo 2
SSD: Samsung 950 Pro (m.2 PCIe) [Benchmark: 1500 MB/s]
SSDs: Highpoint PCIe Raid Controller w/ 8 x OCZ 960 GB SSD [Benchmark: 2500-3000 MB/s]

Studio:
3 x Vaddio UHD PTZ Cameras in 1080p/60
1 x Presenter PC via Blackmagic HDMI > SDI
2 x Projector fed via SDI > HDMI > Cat6 > HDMI
1 x Confidence Monitor fed via SDI > HDMI

Setup:
VMIX System Framerate: 1080P 59.94
The three cameras as well as the PC all plug in to the Kona 4 card and are added as inputs in VMIX as NTSC 1080P 59.94
The Decklink DUO 2 has 2 SDIs as External Output 1 and External Output 2

Usage
The goal is to be able to record the live cut/capture from vMIX in 1080p59.94 as well as use the Multicorder to record each of the incoming camera/PC streams in 1080p59.94.
All three cameras and the Presenter PC all output native 1080p 59.94p.

Problem
When I try and record the capture and all 4 inputs, I drop either a lot of frames, or a few frames depending on the codec, there doesn't seem to be a great solution for dropping no frames.

To try and find the bottleneck, I carefully monitor all the hardware I can, and in many codecs, my hardware is only running at 10% of its capability, yet still dropping frames.

For example:

[img]http://imgur.com/a/n5ieF[/img]
http://imgur.com/a/n5ieF

You can see that my CPU only ran around 39% usage during the whole recording. My Disk only reached around 110MB/s and even then, only occasionally. My Render Time remained between 5ms - 6ms which as far as I know is well within the appropriate range and would indicate that neither my GPU, nor my capture card are reaching their limit. This is true of most of the AVI codecs.

There are two codecs that very rarely drop frames, they are MagicYUV Lossless and Lagarith Lossless. I would gladly take either of the Lossless codecs if they truly never dropped frames, but even they will occasionally drop a group of frames every 15 minutes or so. That being said, I don't want to just find one codec that works while the rest of them inexplicably don't. I want to understand why it is dropping frames at all on some codecs and how to stop it.

There are some codecs that drop frames, and the reason is well known. If I use FFMPEG Pro Res, I max out my CU just after two streams, so obviously it is going to drop frames recording 5 streams simultaneously. But for every codec that it is easy to see why it drops frames, there are 5 that drop frames even at low system usage.

Does anybody have any tips or suggestions as to what I am doing wrong and how I can do it right?

This is a problem with 1080P59.94 but ultimately we want our cameras to record in 4K, both vMix and the Kona4 card claim to support 4K, yet in practice we can't even do 1080P60 right, much less 4k.

Thanks in advance,
VisionCloud

thanks 1 user thanked VisionCloud for this useful post.
sinc747 on 9/22/2016(UTC)
admin  
#2 Posted : Thursday, September 22, 2016 12:02:56 AM(UTC)
admin

Rank: Administration

Groups: Administrators
Joined: 1/13/2010(UTC)
Posts: 5,208
Man
Location: Gold Coast, Australia

Was thanked: 4288 time(s) in 1520 post(s)
Thanks for the detailed information it is very helpful.
In the screenshot it shows the main recording is dropping frames "Due to Hard Disk/CPU"
can you confirm if it is reporting the same in the logs for each MultiCorder recording?

If so that leaves two possibilities:

1. CPU, some codecs will only use one core at a time, so check the CPU usage of each core and see if any are sitting at 100% by themselves.

2. Disk, on the surface this seems unlikely due to the incredibly powerful SSD setup you have running, but there may be an issue somewhere
causing high latency for "random writes" as the average response time in Task Manager is very high for an SSD. (63.2ms)

If you have drive benchmark software available try running a random write test to see what the performance is like.
Also try running one recording, check the response time, then 2, 3, 4 and so on until you see a sudden increase in the response time that coincides with the dropped frames.

There have been reports of PCI Express latency issues on X99 motherboards as well. Usually updating drivers across the board, especially for things like the network devices is enough to fix it.
You can also monitor for problematic drivers using LatencyMon:

http://www.resplendence.com/latencymon

Hope this helps.

Regards,

Martin
vMix

VisionCloud  
#3 Posted : Thursday, September 22, 2016 10:58:51 AM(UTC)
VisionCloud

Rank: Newbie

Groups: Registered
Joined: 9/21/2016(UTC)
Posts: 2
Location: Utah, USA

Was thanked: 1 time(s) in 1 post(s)
Thanks for your response. Those are definitely some good things to look at.

1. That is an interesting thought. I know that Lagarith, one of the ones that rarely drops frames if ever gives me a multithread option in its settings. Grass Valley HQX, one that drops frames (seemingly inexplicably) doesn't say one way or another. When running Grass Valley HQX, all 16 threads hold at that 38 - 40% CPU usage range, none of the individual cores or threads seem to be getting hit harder than the other.

2. I've been benchmarking the SSDs using a handful of tools, and best I can tell, they seem to handle Random Writes okay. For a while, I've been assuming that the random dropped frames I get that only happen rarely on otherwise good codecs are just because the SSD raid hiccupped (very scientific term.) I'm wondering if maybe different SSDs would work better. The ones I hav are relatively cheap OCZs, where I know that Samsung EVO Pros have slightly better performance. I might be able to get 4-8 of those in here in the next few days and give those a test too.

Originally, I had all the SSDs plugged in directly to the MoBo. I was finding that whether software raid, hardware raid or individual drives, the total performance of all drives combined could not surpass 1300 - 1500 MB/s and originally, I assumed that was my problem. That is why I got the PCIE Raid card. I wonder if the limited bandwidth on the SATA would still beat latency on the PCIE Express. Something I could experiment with.

If X99 is going to be a jerk, I could give Z170 or something else a try. I know that the DMI in X99 is only gen 2 and is what limits the SSD performance when using the X99 chipset SATA Controller, Whereas Z170 uses DMI 3.0 which has twice the bandwidth of 2.0

My original worry was that Z170 doesn't have/support as many lanes from the CPU as X99 does. But, it might be a worthwhile tradeoff.

Thanks for latencymon, definitely going to give that a look.

I'll post back any additional findings that I come across.

Thanks,
VisionCloud
jasonqian  
#4 Posted : Monday, October 17, 2016 5:23:38 AM(UTC)
jasonqian

Rank: Newbie

Groups: Registered
Joined: 10/17/2016(UTC)
Posts: 2

Your problem is likely to be in the selection of 2 different brands of the acquisition card;



You can do multiple tests with 1 cards.

Jasonqian 45828484@qq.com
h2video.nl  
#5 Posted : Wednesday, October 19, 2016 3:19:10 AM(UTC)
h2video.nl

Rank: Advanced Member

Groups: Registered
Joined: 3/7/2012(UTC)
Posts: 503
Man
Location: The Netherlands

Thanks: 9 times
Was thanked: 45 time(s) in 39 post(s)
why do you capture such high frame rate? try setting it to 30p or 60i and set system framerate to 30p
stefan
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.