logo

Live Production Software Forums


Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

2 Pages12>
Options
Go to last post Go to first unread
millst  
#1 Posted : Sunday, September 9, 2018 4:20:55 PM(UTC)
millst

Rank: Member

Groups: Registered
Joined: 4/28/2018(UTC)
Posts: 25
Location: NZ

Was thanked: 7 time(s) in 3 post(s)
There has been some discussion about threadripper cpu’s not being suitable for use with vmix.

However, if you want to do high channel count SDI, you really have no choice as you need the 64 PCIe lanes that threadripper boasts.

We decided to bite the bullet and go all out to make it work.

Results below, I’ll add screen shots shortly.

X399 Aorus Motherboard
Threadripper 1950x 16 core CPU with 32 logical processors and 64 PCIE lanes.
32Gb DDR 4
Gigabyte Aorus Nvidia GTX 1080TI Extreme Edition with 11Gb
512Gb Samsung NVME
2 x Decklink Quad 2 cards for a total of 16 SDI inputs
Intel SFP+ card with ubiquitous single mode fibre modules for 10Gbit networking


Results

In vmix at 1080p50 with all 16 SDI inputs running
5% Vmix CPU, 10% total, 10ms latency
GPU at 25%

We still have some more testing to do on end to end latency from SDI in to NDI out which I will post.

It’s mounted in a 4u rack and the plan is to use this as a standalone system for smaller jobs.
For larger jobs this will be the capture machine and will export all channels to a second machine over NDI which will do the mixing.
Because of the high channel count it needs 10Gbit so we went with fibre to allow us some flexibility on distance.

For example, at a concert we can have this machine on stage with all the cameras connected to it then run a single fibre out to our OB van where we do the mix. This keeps the setup nice and simple.

Let me know if you have any questions or tests you want me to run. We are pretty happy with the performance of this machine. It eats out intel 7700 machines for breakfast on high workloads like this and the high number of PCIe lanes means we could in theory add another decklink card for 24 inputs.


thanks 4 users thanked millst for this useful post.
Peter B on 9/10/2018(UTC), SportsNetUSA.net on 9/11/2018(UTC), PFBM on 11/4/2018(UTC), nikosman88 on 3/24/2022(UTC)
richardgatarski  
#2 Posted : Sunday, September 9, 2018 5:40:15 PM(UTC)
richardgatarski

Rank: Advanced Member

Groups: Registered
Joined: 2/18/2014(UTC)
Posts: 1,811
Location: Stockholm

Thanks: 137 times
Was thanked: 292 time(s) in 246 post(s)
First of all, thanks a lot for sharing!

No offense, but were all SDI sources from cameras with live action (and not static shots which are not so demanding for the hardware)?

Did you do any kind of recording/streaming?

What Statistics did you get (colorful button lower right in vMix's window)?

Considering that de/encoding NDI requires a fair amount of CPU power, it will be interesting to see what happens when you add that.

Keep up the great build :)


millst  
#3 Posted : Monday, September 10, 2018 5:08:30 AM(UTC)
millst

Rank: Member

Groups: Registered
Joined: 4/28/2018(UTC)
Posts: 25
Location: NZ

Was thanked: 7 time(s) in 3 post(s)
Hi Richard

We were running a countdown timer so you can measure latency.
Screenshots attached.

millst attached the following image(s):
2018-09-09 21_03_59-Window.png (272kb) downloaded 84 time(s).
2018-09-09 21_04_25-Window.png (5kb) downloaded 72 time(s).
2018-09-09 21_04_57-Task Manager.png (30kb) downloaded 76 time(s).
2018-09-09 21_05_09-Blackmagic Desktop Video Setup.png (75kb) downloaded 68 time(s).
2018-09-09 21_05_37-Task Manager.png (21kb) downloaded 56 time(s).

You cannot view/download attachments. Try to login or register.
millst  
#4 Posted : Monday, September 10, 2018 5:10:06 AM(UTC)
millst

Rank: Member

Groups: Registered
Joined: 4/28/2018(UTC)
Posts: 25
Location: NZ

Was thanked: 7 time(s) in 3 post(s)
more screen shots
millst attached the following image(s):
2018-09-09 21_03_59-Window.png (272kb) downloaded 43 time(s).
2018-09-09 21_07_51-Statistics.png (37kb) downloaded 41 time(s).

You cannot view/download attachments. Try to login or register.
millst  
#5 Posted : Monday, September 10, 2018 5:13:00 AM(UTC)
millst

Rank: Member

Groups: Registered
Joined: 4/28/2018(UTC)
Posts: 25
Location: NZ

Was thanked: 7 time(s) in 3 post(s)
We are still doing a bit more testing with the decklink cards in different slots as there are a few dropped frames there. Normally we run in 25p so at this point we are really just stressing it to see if it breaks. It looks like the decklink cards are saturating their own 8 pcie lanes. I'll do some testing on this front to confirm.
JoseL  
#6 Posted : Monday, September 10, 2018 1:00:02 PM(UTC)
JoseL

Rank: Advanced Member

Groups: Registered
Joined: 4/15/2018(UTC)
Posts: 64
Man
Location: Spain

Thanks: 6 times
Was thanked: 19 time(s) in 13 post(s)
If you want to test correctly need to send a good test pattern, i usually play noise plus a horizontal moving bar (to check it is fluid) to stress system to the limit.
thanks 1 user thanked JoseL for this useful post.
TechnicalMonkey on 9/29/2018(UTC)
millst  
#7 Posted : Monday, September 10, 2018 5:06:20 PM(UTC)
millst

Rank: Member

Groups: Registered
Joined: 4/28/2018(UTC)
Posts: 25
Location: NZ

Was thanked: 7 time(s) in 3 post(s)
some more digging and one of the decklink cards is sharing pcie lanes with an SSD so this is dropping frames and increasing latency on one of the cards.
We also tried changing to a cinema test pattern and it had no appreciable difference on load.
We tried pulling 16 NDI sources from it and it almost maxes out the GPU but it doesn't drop any NDI frames.
Again this is at 50p so a higher frame rate than we will actually run.
millst attached the following image(s):
2018-09-10 09_07_20-Window.png (1,022kb) downloaded 79 time(s).
2018-09-10 09_08_03-Statistics.png (39kb) downloaded 50 time(s).

You cannot view/download attachments. Try to login or register.
millst  
#8 Posted : Monday, September 10, 2018 5:10:41 PM(UTC)
millst

Rank: Member

Groups: Registered
Joined: 4/28/2018(UTC)
Posts: 25
Location: NZ

Was thanked: 7 time(s) in 3 post(s)
we also tried a multi-corder on 16 SDI sources, this maxes out the GPU with the h264 encode, the most we can get to is 12 multicorders at 50p.
mjgraves  
#9 Posted : Monday, September 10, 2018 5:58:23 PM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
millst wrote:
we also tried a multi-corder on 16 SDI sources, this maxes out the GPU with the h264 encode, the most we can get to is 12 multicorders at 50p.


Interesting. What GPU? It occurs to me that the right choice of GPU would deliver unlocked nvenc that could push that further.
ask  
#10 Posted : Tuesday, September 11, 2018 12:13:02 AM(UTC)
ask

Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 1,161
Australia
Location: Melbourne

Thanks: 220 times
Was thanked: 199 time(s) in 181 post(s)
mjgraves wrote:
millst wrote:
we also tried a multi-corder on 16 SDI sources, this maxes out the GPU with the h264 encode, the most we can get to is 12 multicorders at 50p.


Interesting. What GPU? It occurs to me that the right choice of GPU would deliver unlocked nvenc that could push that further.


Gigabyte Aorus Nvidia GTX 1080TI Extreme Edition with 11Gb


This card only allows two encodes at one time.


This from vMix "....the Quadro P series supports many more simultaneous encodes than 2. (No fixed limit, up to what the card is capable of)
Other than that performance is equal to a GeForce with the same core count"


JoseL  
#11 Posted : Tuesday, September 11, 2018 6:00:01 AM(UTC)
JoseL

Rank: Advanced Member

Groups: Registered
Joined: 4/15/2018(UTC)
Posts: 64
Man
Location: Spain

Thanks: 6 times
Was thanked: 19 time(s) in 13 post(s)
12 multicorders at 1080 50p is a lot!,

cpu at 100%?, what latency?.

What compresor and bitrate?. H264 at 75mbps?.
TechnicalMonkey  
#12 Posted : Saturday, September 15, 2018 8:53:45 AM(UTC)
TechnicalMonkey

Rank: Newbie

Groups: Registered
Joined: 9/8/2018(UTC)
Posts: 5
Location: New York

Thanks: 2 times
I have a similar setup, but I'm using Datapath cards instead. I have 2 3G-SDI and 4 DVI-I ports, and I would love to stress test the heck out of them and see if I can still get all of this to work with CPU encoding at 1080p60 using a slow preset on FFMPEG.

@millst: I think you can get a lot more stable performance if you lock in the max CPU rates. I usually have mine locked in at 3.7GHz. I don't care about slowing it down, I need it to perform. That is the attitude I take with my hardware. I do similar things to the GPU as well.

@JoseL or @richardgatarski: do you guys know of stable noise patterns (ones without a single frame drop) at 1080p60 that I can find? I would love to stress test my rig.

Lets keep this interesting conversation going.
mjgraves  
#13 Posted : Saturday, September 15, 2018 1:59:12 PM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
According to this: https://developer.nvidia...ecode-gpu-support-matrix

The GTX 1080 Ti has only two physical encoders, each restricted to one session.

Using a Quadro P2000 provides 2 physical instances of nvenc, without restriction of the number of sessions.

Using a Quadro P5000 provides 3 physical instances of nvenc, without restriction of the number of sessions.

JoseL  
#14 Posted : Saturday, September 15, 2018 5:29:49 PM(UTC)
JoseL

Rank: Advanced Member

Groups: Registered
Joined: 4/15/2018(UTC)
Posts: 64
Man
Location: Spain

Thanks: 6 times
Was thanked: 19 time(s) in 13 post(s)
P2000 and P4000 have only 1 physical nvidia encoder.
P5000 and P6000 have 2 physical encoder

With p4000 can encode about 6-7 streams at 1080 50p. I order a p5000, maybe next weeks i test between p4000 and p5000.

mjgraves wrote:
According to this: https://developer.nvidia...ecode-gpu-support-matrix

The GTX 1080 Ti has only two physical encoders, each restricted to one session.

Using a Quadro P2000 provides 2 physical instances of nvenc, without restriction of the number of sessions.

Using a Quadro P5000 provides 3 physical instances of nvenc, without restriction of the number of sessions.


thanks 1 user thanked JoseL for this useful post.
mjgraves on 11/12/2019(UTC)
Vuurmannetje  
#15 Posted : Sunday, September 16, 2018 4:34:41 PM(UTC)
Vuurmannetje

Rank: Advanced Member

Groups: Registered
Joined: 5/14/2018(UTC)
Posts: 112
Location: Netherlands

Thanks: 3 times
Was thanked: 28 time(s) in 18 post(s)
Thanks a lot for sharing your system!

This makes me feel a lot more confident with my own leap of faith, although Im going for the 2990WX myself.

Indeed on the Multicorders maxing out GPU, the hardware encoders only really offload it. I wonder if you can get more going with a second video card.
I believe there are some older Quadros you can get with loads of Encoders as well for low price, if you were to build a pure recording device.

I went to IBC today and talked to Tim a bit about AMD, and the main reason for not testing on AMD was the PCI lane latency the first series of TR and Ryzen had. Your results reinforce my idea that they had it fixed in series 2, as this latency was also a major cause of bad gaming benchmarks for TR1.

Ive got the parts in today, so later in the next week ill be testing my system as well.

mjgraves  
#16 Posted : Sunday, September 16, 2018 9:05:25 PM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
JoseL wrote:
P2000 and P4000 have only 1 physical nvidia encoder.
P5000 and P6000 have 2 physical encoder

With p4000 can encode about 6-7 streams at 1080 50p. I order a p5000, maybe next weeks i test between p4000 and p5000.


Yes, I misread the table. The GV100 has 3 physical encoders, but at a whopping $24k!
Vuurmannetje  
#17 Posted : Monday, September 17, 2018 6:05:55 AM(UTC)
Vuurmannetje

Rank: Advanced Member

Groups: Registered
Joined: 5/14/2018(UTC)
Posts: 112
Location: Netherlands

Thanks: 3 times
Was thanked: 28 time(s) in 18 post(s)
How would the lower spec Quadros perform for purely multicorder purposes? And can we use multiple cards to up the count?
JoseL  
#18 Posted : Monday, September 17, 2018 8:48:13 AM(UTC)
JoseL

Rank: Advanced Member

Groups: Registered
Joined: 4/15/2018(UTC)
Posts: 64
Man
Location: Spain

Thanks: 6 times
Was thanked: 19 time(s) in 13 post(s)
I think this is a bad idea.

1. You can not mix quadro a geforce
2. Very poor performance on old quadro. I tested k4200 performance on encoding was really bad. New p4000 is not so expensive and performance is great.
3. You will loose pcie lanes and slots
thanks 2 users thanked JoseL for this useful post.
Vuurmannetje on 9/17/2018(UTC), mjgraves on 9/17/2018(UTC)
lael  
#19 Posted : Saturday, October 13, 2018 2:01:30 PM(UTC)
lael

Rank: Advanced Member

Groups: Registered
Joined: 2/18/2017(UTC)
Posts: 75

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
How are these systems perfoming? Has anyone run their system with real world multiple NDI inputs into vMix?
Ittaidv  
#20 Posted : Monday, October 29, 2018 11:08:10 PM(UTC)
Ittaidv

Rank: Advanced Member

Groups: Registered
Joined: 12/19/2013(UTC)
Posts: 600
Man
Belgium
Location: Belgium

Thanks: 75 times
Was thanked: 91 time(s) in 75 post(s)
I personaly use x299 with asus ws motherboards, they use plx chips on their motherboard, so the amount of lanes of the intel 7900x is doubled. I get better results than these stats in terms of lost frames, and can do the same amount of multicorders. I understand the idea of going for threadripper, but don't think it's the best idea if you really need high performance.
Users browsing this topic
2 Pages12>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.