vMix - Live Production Software
Live Production Software Forums       www.vmix.com
Welcome Guest Search | Active Topics | Log In | Register

Tag as favorite
Ultimate VMix machine using threadripper
millst
#1 Posted : Sunday, 9 September 2018 6:20:55 AM
Rank: Newbie

Medals:

Groups: Registered
Joined: 27/04/2018
Posts: 8
Location: NZ

Thanks: 0 times
Was thanked: 2 time(s) in 1 post(s)
There has been some discussion about threadripper cpu’s not being suitable for use with vmix.

However, if you want to do high channel count SDI, you really have no choice as you need the 64 PCIe lanes that threadripper boasts.

We decided to bite the bullet and go all out to make it work.

Results below, I’ll add screen shots shortly.

X399 Aorus Motherboard
Threadripper 1950x 16 core CPU with 32 logical processors and 64 PCIE lanes.
32Gb DDR 4
Gigabyte Aorus Nvidia GTX 1080TI Extreme Edition with 11Gb
512Gb Samsung NVME
2 x Decklink Quad 2 cards for a total of 16 SDI inputs
Intel SFP+ card with ubiquitous single mode fibre modules for 10Gbit networking


Results

In vmix at 1080p50 with all 16 SDI inputs running
5% Vmix CPU, 10% total, 10ms latency
GPU at 25%

We still have some more testing to do on end to end latency from SDI in to NDI out which I will post.

It’s mounted in a 4u rack and the plan is to use this as a standalone system for smaller jobs.
For larger jobs this will be the capture machine and will export all channels to a second machine over NDI which will do the mixing.
Because of the high channel count it needs 10Gbit so we went with fibre to allow us some flexibility on distance.

For example, at a concert we can have this machine on stage with all the cameras connected to it then run a single fibre out to our OB van where we do the mix. This keeps the setup nice and simple.

Let me know if you have any questions or tests you want me to run. We are pretty happy with the performance of this machine. It eats out intel 7700 machines for breakfast on high workloads like this and the high number of PCIe lanes means we could in theory add another decklink card for 24 inputs.


2 users thanked millst for this useful post.
Peter B on 10/09/2018, SportsNetUSA.net on 10/09/2018
richardgatarski
#2 Posted : Sunday, 9 September 2018 7:40:15 AM
Rank: Advanced Member

Medals:

Groups: Registered
Joined: 18/02/2014
Posts: 1,235
Location: Stockholm

Thanks: 103 times
Was thanked: 189 time(s) in 157 post(s)
First of all, thanks a lot for sharing!

No offense, but were all SDI sources from cameras with live action (and not static shots which are not so demanding for the hardware)?

Did you do any kind of recording/streaming?

What Statistics did you get (colorful button lower right in vMix's window)?

Considering that de/encoding NDI requires a fair amount of CPU power, it will be interesting to see what happens when you add that.

Keep up the great build :)


millst
#3 Posted : Sunday, 9 September 2018 7:08:30 PM
Rank: Newbie

Medals:

Groups: Registered
Joined: 27/04/2018
Posts: 8
Location: NZ

Thanks: 0 times
Was thanked: 2 time(s) in 1 post(s)
Hi Richard

We were running a countdown timer so you can measure latency.
Screenshots attached.

File Attachment(s):
2018-09-09 21_03_59-Window.png (272kb) downloaded 0 time(s).
millst attached the following image(s):
2018-09-09 21_04_25-Window.png (5kb) downloaded 0 time(s).
2018-09-09 21_04_57-Task Manager.png (30kb) downloaded 0 time(s).
2018-09-09 21_05_09-Blackmagic Desktop Video Setup.png (75kb) downloaded 0 time(s).
2018-09-09 21_05_37-Task Manager.png (21kb) downloaded 0 time(s).
millst
#4 Posted : Sunday, 9 September 2018 7:10:06 PM
Rank: Newbie

Medals:

Groups: Registered
Joined: 27/04/2018
Posts: 8
Location: NZ

Thanks: 0 times
Was thanked: 2 time(s) in 1 post(s)
more screen shots
File Attachment(s):
2018-09-09 21_03_59-Window.png (272kb) downloaded 3 time(s).
millst attached the following image(s):
2018-09-09 21_07_51-Statistics.png (37kb) downloaded 0 time(s).
millst
#5 Posted : Sunday, 9 September 2018 7:13:00 PM
Rank: Newbie

Medals:

Groups: Registered
Joined: 27/04/2018
Posts: 8
Location: NZ

Thanks: 0 times
Was thanked: 2 time(s) in 1 post(s)
We are still doing a bit more testing with the decklink cards in different slots as there are a few dropped frames there. Normally we run in 25p so at this point we are really just stressing it to see if it breaks. It looks like the decklink cards are saturating their own 8 pcie lanes. I'll do some testing on this front to confirm.
JoseL
#6 Posted : Monday, 10 September 2018 3:00:02 AM
Rank: Member

Medals:

Groups: Registered
Joined: 14/04/2018
Posts: 23
Location: Spain

Thanks: 4 times
Was thanked: 6 time(s) in 3 post(s)
If you want to test correctly need to send a good test pattern, i usually play noise plus a horizontal moving bar (to check it is fluid) to stress system to the limit.
millst
#7 Posted : Monday, 10 September 2018 7:06:20 AM
Rank: Newbie

Medals:

Groups: Registered
Joined: 27/04/2018
Posts: 8
Location: NZ

Thanks: 0 times
Was thanked: 2 time(s) in 1 post(s)
some more digging and one of the decklink cards is sharing pcie lanes with an SSD so this is dropping frames and increasing latency on one of the cards.
We also tried changing to a cinema test pattern and it had no appreciable difference on load.
We tried pulling 16 NDI sources from it and it almost maxes out the GPU but it doesn't drop any NDI frames.
Again this is at 50p so a higher frame rate than we will actually run.
File Attachment(s):
2018-09-10 09_07_20-Window.png (1,022kb) downloaded 1 time(s).
millst attached the following image(s):
2018-09-10 09_08_03-Statistics.png (39kb) downloaded 2 time(s).
millst
#8 Posted : Monday, 10 September 2018 7:10:41 AM
Rank: Newbie

Medals:

Groups: Registered
Joined: 27/04/2018
Posts: 8
Location: NZ

Thanks: 0 times
Was thanked: 2 time(s) in 1 post(s)
we also tried a multi-corder on 16 SDI sources, this maxes out the GPU with the h264 encode, the most we can get to is 12 multicorders at 50p.
mjgraves
#9 Posted : Monday, 10 September 2018 7:58:23 AM
Rank: Advanced Member

Medals:

Groups: Registered
Joined: 1/07/2015
Posts: 632
Location: Houston TX

Thanks: 201 times
Was thanked: 152 time(s) in 130 post(s)
millst wrote:
we also tried a multi-corder on 16 SDI sources, this maxes out the GPU with the h264 encode, the most we can get to is 12 multicorders at 50p.


Interesting. What GPU? It occurs to me that the right choice of GPU would deliver unlocked nvenc that could push that further.
ask
#10 Posted : Monday, 10 September 2018 2:13:02 PM
Rank: Advanced Member

Medals:

Groups: Registered
Joined: 12/10/2012
Posts: 737
Location: Melbourne

Thanks: 113 times
Was thanked: 133 time(s) in 121 post(s)
mjgraves wrote:
millst wrote:
we also tried a multi-corder on 16 SDI sources, this maxes out the GPU with the h264 encode, the most we can get to is 12 multicorders at 50p.


Interesting. What GPU? It occurs to me that the right choice of GPU would deliver unlocked nvenc that could push that further.


Gigabyte Aorus Nvidia GTX 1080TI Extreme Edition with 11Gb


This card only allows two encodes at one time.


This from vMix "....the Quadro P series supports many more simultaneous encodes than 2. (No fixed limit, up to what the card is capable of)
Other than that performance is equal to a GeForce with the same core count"


JoseL
#11 Posted : Monday, 10 September 2018 8:00:01 PM
Rank: Member

Medals:

Groups: Registered
Joined: 14/04/2018
Posts: 23
Location: Spain

Thanks: 4 times
Was thanked: 6 time(s) in 3 post(s)
12 multicorders at 1080 50p is a lot!,

cpu at 100%?, what latency?.

What compresor and bitrate?. H264 at 75mbps?.
TechnicalMonkey
#12 Posted : Friday, 14 September 2018 10:53:45 PM
Rank: Newbie

Medals:

Groups: Registered
Joined: 8/09/2018
Posts: 2
Location: New York

Thanks: 1 times
Was thanked: 0 time(s) in 0 post(s)
I have a similar setup, but I'm using Datapath cards instead. I have 2 3G-SDI and 4 DVI-I ports, and I would love to stress test the heck out of them and see if I can still get all of this to work with CPU encoding at 1080p60 using a slow preset on FFMPEG.

@millst: I think you can get a lot more stable performance if you lock in the max CPU rates. I usually have mine locked in at 3.7GHz. I don't care about slowing it down, I need it to perform. That is the attitude I take with my hardware. I do similar things to the GPU as well.

@JoseL or @richardgatarski: do you guys know of stable noise patterns (ones without a single frame drop) at 1080p60 that I can find? I would love to stress test my rig.

Lets keep this interesting conversation going.
mjgraves
#13 Posted : Saturday, 15 September 2018 3:59:12 AM
Rank: Advanced Member

Medals:

Groups: Registered
Joined: 1/07/2015
Posts: 632
Location: Houston TX

Thanks: 201 times
Was thanked: 152 time(s) in 130 post(s)
According to this: https://developer.nvidia...code-gpu-support-matrix

The GTX 1080 Ti has only two physical encoders, each restricted to one session.

Using a Quadro P2000 provides 2 physical instances of nvenc, without restriction of the number of sessions.

Using a Quadro P5000 provides 3 physical instances of nvenc, without restriction of the number of sessions.

JoseL
#14 Posted : Saturday, 15 September 2018 7:29:49 AM
Rank: Member

Medals:

Groups: Registered
Joined: 14/04/2018
Posts: 23
Location: Spain

Thanks: 4 times
Was thanked: 6 time(s) in 3 post(s)
P2000 and P4000 have only 1 physical nvidia encoder.
P5000 and P6000 have 2 physical encoder

With p4000 can encode about 6-7 streams at 1080 50p. I order a p5000, maybe next weeks i test between p4000 and p5000.

mjgraves wrote:
According to this: https://developer.nvidia...code-gpu-support-matrix

The GTX 1080 Ti has only two physical encoders, each restricted to one session.

Using a Quadro P2000 provides 2 physical instances of nvenc, without restriction of the number of sessions.

Using a Quadro P5000 provides 3 physical instances of nvenc, without restriction of the number of sessions.


Vuurmannetje
#15 Posted : Sunday, 16 September 2018 6:34:41 AM
Rank: Advanced Member

Medals:

Groups: Registered
Joined: 13/05/2018
Posts: 36
Location: Netherlands

Thanks: 2 times
Was thanked: 7 time(s) in 4 post(s)
Thanks a lot for sharing your system!

This makes me feel a lot more confident with my own leap of faith, although Im going for the 2990WX myself.

Indeed on the Multicorders maxing out GPU, the hardware encoders only really offload it. I wonder if you can get more going with a second video card.
I believe there are some older Quadros you can get with loads of Encoders as well for low price, if you were to build a pure recording device.

I went to IBC today and talked to Tim a bit about AMD, and the main reason for not testing on AMD was the PCI lane latency the first series of TR and Ryzen had. Your results reinforce my idea that they had it fixed in series 2, as this latency was also a major cause of bad gaming benchmarks for TR1.

Ive got the parts in today, so later in the next week ill be testing my system as well.

mjgraves
#16 Posted : Sunday, 16 September 2018 11:05:25 AM
Rank: Advanced Member

Medals:

Groups: Registered
Joined: 1/07/2015
Posts: 632
Location: Houston TX

Thanks: 201 times
Was thanked: 152 time(s) in 130 post(s)
JoseL wrote:
P2000 and P4000 have only 1 physical nvidia encoder.
P5000 and P6000 have 2 physical encoder

With p4000 can encode about 6-7 streams at 1080 50p. I order a p5000, maybe next weeks i test between p4000 and p5000.


Yes, I misread the table. The GV100 has 3 physical encoders, but at a whopping $24k!
Vuurmannetje
#17 Posted : Sunday, 16 September 2018 8:05:55 PM
Rank: Advanced Member

Medals:

Groups: Registered
Joined: 13/05/2018
Posts: 36
Location: Netherlands

Thanks: 2 times
Was thanked: 7 time(s) in 4 post(s)
How would the lower spec Quadros perform for purely multicorder purposes? And can we use multiple cards to up the count?
JoseL
#18 Posted : Sunday, 16 September 2018 10:48:13 PM
Rank: Member

Medals:

Groups: Registered
Joined: 14/04/2018
Posts: 23
Location: Spain

Thanks: 4 times
Was thanked: 6 time(s) in 3 post(s)
I think this is a bad idea.

1. You can not mix quadro a geforce
2. Very poor performance on old quadro. I tested k4200 performance on encoding was really bad. New p4000 is not so expensive and performance is great.
3. You will loose pcie lanes and slots
2 users thanked JoseL for this useful post.
Vuurmannetje on 17/09/2018, mjgraves on 17/09/2018
Users browsing this topic
Guest
Tag as favorite
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAFPro Theme Created by Jaben Cargman (Tiny Gecko)
Powered by YAF | YAF © 2003-2009, Yet Another Forum.NET
This page was generated in 0.119 seconds.