logo

Live Production Software Forums


Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
Geoff B  
#1 Posted : Wednesday, July 1, 2020 1:27:50 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Two shows in a row, I've had major problems with audio dropping out on NDI sources between vMix instances. These are running on AWS in the same subnet of the same VPC.

On one show, we tracked the problem down to too much NDI network traffic. As soon as we closed a few VPN-connected streams, the problem cleared up. We had big audio dropouts on NDI video sources and audio-only NDI sources.

My very next show, we had huge audio dropouts again. Again, it affected both A/V and audio-only sources. But this time, it was a very small show between two instances. We never figured out the cause, and we suffered with bad audio for over 4 hours.

Any ideas of what I can do to reduce the fragility of NDI?
mavik  
#2 Posted : Wednesday, July 1, 2020 5:12:17 PM(UTC)
mavik

Rank: Advanced Member

Groups: Registered
Joined: 4/23/2017(UTC)
Posts: 421
Man
Location: Germany

Thanks: 2 times
Was thanked: 68 time(s) in 63 post(s)
Have you tried SRT instead of NDI. That should add a bit of security.
Most likely this is network related. Can you setup QoS rules and prioritize NDI traffic.
thanks 1 user thanked mavik for this useful post.
Geoff B on 7/4/2020(UTC)
Geoff B  
#3 Posted : Wednesday, July 1, 2020 11:08:11 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Mavik, thanks so much for your insight.

Originally Posted by: mavik Go to Quoted Post
Have you tried SRT instead of NDI. That should add a bit of security.


I haven't tried this. I can certainly do that for the sources that are outputs. Calls and audio busses will have to remain NDI, however.

Originally Posted by: mavik Go to Quoted Post
Can you setup QoS rules and prioritize NDI traffic.


An interesting question. I will have to dig in on AWS and see. But I can't imagine on my second show I was anywhere near saturating the subnet (the machines have 25Gbps NICs).

Again, I appreciate your reply! Thank you!
Geoff B  
#4 Posted : Friday, July 3, 2020 3:21:47 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Update:

Had another show last night that was the same setup as the last one. Re-shuffled the show quite a bit to reduce dependency on NDI, but we were still having NDI audio dropouts. At this point, there were only 3 NDI video sources in use, and 4 audio busses. I moved the only thing I could to SRT, and that cleared up the issues. Not sure why I'm having so much trouble.
jb-uk  
#5 Posted : Monday, July 6, 2020 4:49:51 AM(UTC)
jb-uk

Rank: Newbie

Groups: Registered
Joined: 6/6/2020(UTC)
Posts: 2
United Kingdom
Location: London

Hi Geoff - any update or progress on this for you? We're planning a similar setup in AWS.

JB
mavik  
#6 Posted : Monday, July 6, 2020 5:01:17 AM(UTC)
mavik

Rank: Advanced Member

Groups: Registered
Joined: 4/23/2017(UTC)
Posts: 421
Man
Location: Germany

Thanks: 2 times
Was thanked: 68 time(s) in 63 post(s)
On the newtek site there is a NDIAnalysis app available. Maybe you get more insight with that tool. I haven't used it myself so far but might be an option.

And second, if possible on AWS, switch NDI from unicast to multicast. Multicast requires IGMP snooping. I don't know if that is something AWS provides.
thanks 1 user thanked mavik for this useful post.
Geoff B on 7/6/2020(UTC)
Geoff B  
#7 Posted : Monday, July 6, 2020 5:05:47 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: jb-uk Go to Quoted Post
Hi Geoff - any update or progress on this for you? We're planning a similar setup in AWS.

JB


Nothing new to report. Hoping to do some testing midweek with someone who knows networking better than me.

I can tell you that I have successfully had 5 vMix instances running on AWS with the main machine having 9 NDI video sources in and 6 out (plus 7 audio busses in & out). But I've also had failures with far smaller operations, so I would like to have a better understanding of what causes it to break.

DWAM  
#8 Posted : Monday, July 6, 2020 6:11:55 AM(UTC)
DWAM

Rank: Advanced Member

Groups: Registered
Joined: 3/20/2014(UTC)
Posts: 2,418
Man
France
Location: Bordeaux, France

Thanks: 198 times
Was thanked: 686 time(s) in 527 post(s)
Using NDI for audio only sources is a non sense

Use audio oriented IP protocols like Dante or VBAN, better quality, better optimization, less ressource...
Geoff B  
#9 Posted : Monday, July 6, 2020 7:18:10 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: mavik Go to Quoted Post
On the newtek site there is a NDIAnalysis app available. Maybe you get more insight with that tool. I haven't used it myself so far but might be an option.

And second, if possible on AWS, switch NDI from unicast to multicast. Multicast requires IGMP snooping. I don't know if that is something AWS provides.


Thanks for the tip on the analysis app. I'm going to check that out for sure.

In the U.S., AWS offers multicast only in the N. Virginia data center. My big show is migrating there for that very reason.

Geoff B  
#10 Posted : Monday, July 6, 2020 7:31:47 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: DWAM Go to Quoted Post
Using NDI for audio only sources is a non sense

Use audio oriented IP protocols like Dante or VBAN, better quality, better optimization, less ressource...



Thanks for your suggestion. I used NDI for audio-only sources because that's what vMix generates when you turn them on in the Outputs/NDI/SRT menu. I need to ship many of my audio busses to other instances of vMix on the same network, and this seemed like a simple way to do that.

I'd love to use Dante. I've looked in the documentation, but don't see any reference to it or VBAN. I assume I would have to do something like a Dante Virtual Soundcard but it's unclear if I can assign each bus (Master, Headphones, A-G) to a separate Dante stream. Have you done this before?
DWAM  
#11 Posted : Monday, July 6, 2020 3:12:29 PM(UTC)
DWAM

Rank: Advanced Member

Groups: Registered
Joined: 3/20/2014(UTC)
Posts: 2,418
Man
France
Location: Bordeaux, France

Thanks: 198 times
Was thanked: 686 time(s) in 527 post(s)
Quote:
I've looked in the documentation

Which one ?
mavik  
#12 Posted : Monday, July 6, 2020 7:36:30 PM(UTC)
mavik

Rank: Advanced Member

Groups: Registered
Joined: 4/23/2017(UTC)
Posts: 421
Man
Location: Germany

Thanks: 2 times
Was thanked: 68 time(s) in 63 post(s)
I made a little video a few weeks ago.
thanks 1 user thanked mavik for this useful post.
Geoff B on 7/6/2020(UTC)
mgrayeb  
#13 Posted : Monday, July 6, 2020 9:22:37 PM(UTC)
mgrayeb

Rank: Advanced Member

Groups: Registered
Joined: 3/3/2018(UTC)
Posts: 101

Thanks: 43 times
Was thanked: 3 time(s) in 3 post(s)
Not sure if this is (tangentially) related or not. I too have had dropouts of audio using NDI between two vMix instances on two laptops connected by hardwire to a gigabit switch on the same network over the last few weeks. To be clear, I am not using AWS, but rather, a LAN.

In my case, on the first computer, I'm doing switching, streaming and encoding with graphics and a vMix Call or two and on the second computer, I am bringing in a Zoom meeting using Virtual Audio Cables on both computers as outlined in the instructions on the vMix website for using two computers to bring in a Zoom call.

Both laptops have very similar specs -- the primary one for streaming, encoding and switching has an i7-9750 CPU, 32DB RAM, nVidia 2060 GPU and the second one has an i7-8750H CPU, 16GB RAM, nVidia 1060 gpu -- and I'm running separate instances of vMix on each laptop.

The audio works for a while without a hitch, but then suddenly, I'll get a dropout for like a minute where the remote guest on Zoom can't hear any audio from the master production, or vice versa, or both. It's very disconcerting. I'm using an unmanaged 5 port TP-Link Gigabit switch and I'm providing video Output from the first laptop and Bus B audio from the first laptop to the second laptop via NDI. In return, I'm making the Zoom video (two screens -- one of the guest and one of their PowerPoint), as well as the guest's audio, available via NDI.

I've experienced this issue several times and have been working to try to identify the source of the problem. I posted a note on the NDI Facebook group because I suspected the issue was possibly related to NDI (and, more specifically, possibly related to the switch I'm using) and Kane Peterson there mentioned that he's using a switch that is a slight upgrade from the one I'm using and that it's worked well for him. I realize this won't help you in an AWS situation but thought I'd mention it here. I just purchased the switch he suggested (model TL-SG105E) and will be testing it very soon and will report back here on what I find. I had been using model TL-SG105.

In comparing the feature descriptions for the two switches, the TL-SG105E mentions "IGMP Snooping optimizes multicast applications" whereas the feature description for the model TL-SG105 mentions "Supports 802.1p/DSCP QoS and IGMP Snooping". Not sure if there's a difference between the IGMP Snooping capabilities between the two units (or what that means, for that matter).
thanks 1 user thanked mgrayeb for this useful post.
Geoff B on 7/6/2020(UTC)
Geoff B  
#14 Posted : Monday, July 6, 2020 11:29:34 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: DWAM Go to Quoted Post
Which one ?


vMix documentation. A search for "dante" yields no results.





Geoff B  
#15 Posted : Monday, July 6, 2020 11:37:40 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: mavik Go to Quoted Post
I made a little video a few weeks ago.


Thank you! Very helpful!
Geoff B  
#16 Posted : Wednesday, July 8, 2020 11:52:21 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
UPDATE

I did some testing yesterday. Watching the GPU graph in the Task Manager, I have found that the audio dropouts coincide with spikes in GPU usage, both small and large. The spikes don't have to take the GPU usage to 100%, but the higher the GPU usage before the spike, the more likely a dropout would occur. These spikes can be triggered by doing things in the vMix interface, especially menus that open on top (audio settings, etc).

Although it's difficult to quantify, it appears that GPU strain on the receiving side is more likely to cause dropouts than on the transmitting side.

I have done everything suggested on the Optimising Performance page.

Does anyone have any other suggestions?
mavik  
#17 Posted : Thursday, July 9, 2020 12:06:28 AM(UTC)
mavik

Rank: Advanced Member

Groups: Registered
Joined: 4/23/2017(UTC)
Posts: 421
Man
Location: Germany

Thanks: 2 times
Was thanked: 68 time(s) in 63 post(s)
Can it be that your PCIe lanes are the problem. Usually a GPU spike should not have any results but if the data can't be transfered internally who knows what the symptoms would look like.
Geoff B  
#18 Posted : Thursday, July 9, 2020 12:28:38 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: mavik Go to Quoted Post
Can it be that your PCIe lanes are the problem. Usually a GPU spike should not have any results but if the data can't be transfered internally who knows what the symptoms would look like.


This is running on AWS virtual machines, so not sure if there's anything I can do... but I will certainly take a look this morning.

Thank you, Mavik. I appreciate the helpful responses you give to me and others!

mavik  
#19 Posted : Thursday, July 9, 2020 1:21:25 AM(UTC)
mavik

Rank: Advanced Member

Groups: Registered
Joined: 4/23/2017(UTC)
Posts: 421
Man
Location: Germany

Thanks: 2 times
Was thanked: 68 time(s) in 63 post(s)
I assume you are using the Quadro setup and not the eGPU. The latter is just connected over the network if I understand the AWS setup correct. That would not be the prefered usage. The Quadro should be the better option. Is Cloudwatch providing some more insights?

Generally sharing is caring. I have learned so much from all here and there and still do everyday. For me it's a no-brainer to pass on my knowledge and get others up and running.
Geoff B  
#20 Posted : Thursday, July 9, 2020 2:41:08 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 62
Man
United States
Location: Sacramento, California

Thanks: 10 times
Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: mavik Go to Quoted Post
I assume you are using the Quadro setup and not the eGPU. The latter is just connected over the network if I understand the AWS setup correct. That would not be the prefered usage. The Quadro should be the better option. Is Cloudwatch providing some more insights?

Generally sharing is caring. I have learned so much from all here and there and still do everyday. For me it's a no-brainer to pass on my knowledge and get others up and running.


Yes, we're using the g4 (attached graphics card) instances for all. Cloudwatch is good for letting us know when we're maxing out the bandwidth or CPU, but doesn't help us with GPU strain.

Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.