logo

Live Production Software Forums


Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
hostmaster@passepartout.net  
#1 Posted : Saturday, September 22, 2018 8:58:40 AM(UTC)
hostmaster@passepartout.net

Rank: Newbie

Groups: Registered
Joined: 9/22/2018(UTC)
Posts: 5
Man
Location: Rimini

Hi
This is my first post in the Forum but it's about a bit more than a year that I'm testing / using vMix

Last year I've been asked to set up a live show for the company I work for. They were using skype like products with poor results.
For the sake of testing, one year ago I've made a pre event dryrun with this setup

1 Main Deck hosting the vMix Calls (at our HQ) where a conference room with 30 seats was listening and 2 people were having a speech
8 joining vMix Call (from hotels all around Italy, Sardinia and Sicily included) where a conference room with 30 seats was listening and 1 people was having a speech
Except for a hardware failure in one of the 8 places (BSOD) and a problem with audio that being listened through loudspeaker had a delay and was feeding back the test was quite successful so I further developed the project.

This year I bought all the stuff and this is the HW config

Main Deck vMix Pro Call Host sending at High 720p 2000Kbps
HP Workstation i7 16GB SSD GeF1080
AKAI APC40 MkII
Focusrite Scarlett 2i2
2 Logitech C930e
2 Mic

8 Remote User vMix Basic HD sending at Default 720p 1200Kbps
HP MicroPC i5 8GB SSD Intel-HD630
Focusrite Scarlett 2i2
2 Logitech C930e
2 Mic
Laser presenter (activating shortcut to cut between the cams)

I've set up A and B audio channel so to keep A locally and B remotely
Mics were plugged in the Focusrite and direct monitor sent to room speakers (no local delay) then sent in B audio channel
Used noise gate and compression and so on to avoid feedback and overload

So here are the problems we faced:
1. On one remote location even if testing the internet connection speed was about 18/18Mbit I had to set to the lowest the video they were sending to keep the connection up and the audio was crackling
2. On a second remote location they were getting the audio but the video was slowing down so that every 5 minutes was about 20/30 seconds out of sync that they had to reconnect to the call to resync

On the first case I've been able to check that ping latency from the remote location to vMixcall server was above 120ms meanwhile to our HQ was less than 20ms
On the second case I've not been able to check anything

I'd really like to troubleshoot this problem since I really need to avoid any of this situation so here I ask for your wisdom

How vMixcall traffic is routed?
How much latency affects the streaming even having more than 10Mbits available?
How can I pre check a location to be sure to not having problems on the day of the event?

Any other suggestion or drilling on the issue is welcomed

And BTW vMix Rules!

Regards
Marco
mjgraves  
#2 Posted : Saturday, September 22, 2018 10:30:19 AM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
Marco,

With that many guests, your bandwidth requirements go up quite sharply. You would be better off using a video conference service instead of vMix Call. That way all the audio/video mixing and would be done on a server in the cloud. Each location would only need bandwidth for one stream in/out.

The VC service is designed for real-time, bi-directional engagement. Audio sync should be perfect all the time.

Consider as an example the weekly World of Live Streaming show that uses Zoom.US.
Vuurmannetje  
#3 Posted : Saturday, September 22, 2018 10:33:58 AM(UTC)
Vuurmannetje

Rank: Advanced Member

Groups: Registered
Joined: 5/14/2018(UTC)
Posts: 112
Location: Netherlands

Thanks: 3 times
Was thanked: 28 time(s) in 18 post(s)
Quote:
How vMixcall traffic is routed?
How much latency affects the streaming even having more than 10Mbits available?
How can I pre check a location to be sure to not having problems on the day of the event?


Im not an official Vmix response, but I do work with Vmixcall daily.

Latency: It depends. It depends mostly on the route the call has to make. Bandwith is only a threshold parameter here. In good conditions and with Low Latency enabled, I average 50-100ms latency on callers. This is similar to what skype/zoom will report in latency.
In some cases, callers in the US have bad luck with ISP throttling video feeds, where their webcam will freeze every few seconds on low latency mode (no audio issue). Normal mode fixes this.

How can I pre-check a location?: Get someone on site testing the equipment with you. That's the only real way.
mjgraves  
#4 Posted : Saturday, September 22, 2018 10:45:41 AM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
hostmaster@passepartout.net wrote:
Hi
How vMixcall traffic is routed?
How much latency affects the streaming even having more than 10Mbits available?
How can I pre check a location to be sure to not having problems on the day of the event?


vMix Call is a WebRTC application. It will try various ways to connect the end-points. If it can connect them directly, in a peer-to-peer fashion, it will do that and latency is minimized.

If it can't connect directly, it will proxy the media through a TURN server. This may be required because end-points are behind restrictive routers or firewalls. The proxy in the signal path increases latency.

One of the advantages of using a video conference service is that they will be on a public IP address and have far-end NAT traversal solutions in place.

As for testing, all you can do is place test calls from the location in question and see what happens.
hostmaster@passepartout.net  
#5 Posted : Saturday, September 22, 2018 10:48:26 AM(UTC)
hostmaster@passepartout.net

Rank: Newbie

Groups: Registered
Joined: 9/22/2018(UTC)
Posts: 5
Man
Location: Rimini

Hi mjgraves

Thanks for your answer

Quote:

With that many guests, your bandwidth requirements go up quite sharply. You would be better off using a video conference service instead of vMix Call.


Bandwidth is not an issue in my case, the company I work for has 130/130Mbit actually and we are going to upgrade it to 500/500Mbit on a month or so
It's not a video conference, is a live production so using a video conference service is not a solution.
By the way changing software after having bought it is not an option nor a fair suggestion

Regards
Marco
mjgraves  
#6 Posted : Saturday, September 22, 2018 11:28:30 AM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
hostmaster@passepartout.net wrote:

It's not a video conference, is a live production so using a video conference service is not a solution.


A video conference IS itself a live production. It's just a particular sort of live event.

VC tools are simply tools. You can use them in many ways. After all, you're using Logitech webcams that were specifically designed for video conferencing.

Incidentally, check those webcams and make sure that they're not in H246 mode. That mode introduces latency as it encodes the video for transit over the USB 2 link. They should be in MJPEG (up to 1080p) or YUY2 (720p) mode.

hostmaster@passepartout.net wrote:

By the way changing software after having bought it is not an option nor a fair suggestion


Who said anything about changing software. I use vMix with video conference services every day. They can be complementary.
DWAM  
#7 Posted : Saturday, September 22, 2018 11:53:57 AM(UTC)
DWAM

Rank: Advanced Member

Groups: Registered
Joined: 3/20/2014(UTC)
Posts: 2,721
Man
France
Location: Bordeaux, France

Thanks: 243 times
Was thanked: 794 time(s) in 589 post(s)
Hi Marco

were the Sicily and Sardinia locations affected by your issues ?

You may have a 18/18 Mbits connection speed to your provider in the street or even to a specific backbone, but connections with islands can be problematic/congested depending on the route and peering agreements available.

If these 2 locations are those you had problems with, I'd bet it's due to their geography and ISP infrastructure being less good as on the continent

Regarding vMixCall and WebRTC, I also quite frequently use vMix for connecting several places together during live events productions. Most of the time it remains in France (up to 8 cities) but from time to time I also have to deal with foreign countries all over the planet. North and South America, Asia, Africa (north or Middle-East), Europe...
So far I must say vMix Call does the job very well. I had issues btw, most of the time due to bad network environnement (despite I always provide a setup roadmap and requirements - some people just don't care and use Wifi or shitty network gear) and lazyness or incompetence of the remote crew, so I understand. Twice I also experienced very bad transmissions, each time remote location was a small island in Asia, once they even tried to use a hotel broadband satellite link. Obviously I had to cancel their participation 'cos even audio was unusable.

In general I can say that up to 500ms latency is not an issue at all.
However the internet is not equal everywhere in terms of quality and routing. Peerings agreements between ISPs is also a real problem especially for AV traffic.

Regarding your issues you must also make sure that no firewall prevented vMixCall to connect directly (Michael mentioned the STUN/TURN alternative which impacts QOS). If you're in a corporate environnement with real network management/security/strategy vMix Call can suffer. Best is to make sure you can use the optimized/direct transmission for each location

Guillaume
DWAM  
#8 Posted : Saturday, September 22, 2018 12:02:01 PM(UTC)
DWAM

Rank: Advanced Member

Groups: Registered
Joined: 3/20/2014(UTC)
Posts: 2,721
Man
France
Location: Bordeaux, France

Thanks: 243 times
Was thanked: 794 time(s) in 589 post(s)
Other things to check are :

- local network infrastructure (using iperf3 for LAN network) > routers, switches, cables, etc... + the average load on the LAN itself and the WAN gateway
- the computers themselves: even a brand new computer can be wrongly set or installed
hostmaster@passepartout.net  
#9 Posted : Saturday, September 22, 2018 12:08:55 PM(UTC)
hostmaster@passepartout.net

Rank: Newbie

Groups: Registered
Joined: 9/22/2018(UTC)
Posts: 5
Man
Location: Rimini

mjgraves wrote:

Incidentally, check those webcams and make sure that they're not in H246 mode.


No they are not in H264. I had 8 different vMix Call with identical hardware and setup and I experienced issues only on 2 so I guess is not a configuration issue

mjgraves wrote:

Who said anything about changing software. I use vMix with video conference services every day. They can be complementary.


Sorry I just misunderstood your suggestion I will check out how the two systems interact

DWAM wrote:

were the Sicily and Sardinia locations affected by your issues ?


No, actually the problem was in Rome and Florence, Sardinia was perfect and Sicily was good enough

DWAM wrote:

Regarding your issues you must also make sure that no firewall prevented vMixCall to connect directly (Michael mentioned the STUN/TURN alternative which impacts QOS). If you're in a corporate environnement with real network management/security/strategy vMix Call can suffer. Best is to make sure you can use the optimized/direct transmission for each location


I work in the IT team inside an AS and ISP so to prevent any inconvenience the vMix Call Host had a Public IP on it and I set up the firewall to accept all incoming traffic.
I kept an eye on the firewall traffic monitor and no STUN packets were dropped, that's why I'm 99,99 sure that the issues are not depending on the vMix Call Host

If anyone from vMix needs the vMix Call ID that were suffering the issue to check on logs I'll provide it. I'm sure they can see deeper than me

Marco
ThomasK  
#10 Posted : Sunday, September 23, 2018 3:39:09 PM(UTC)
ThomasK

Rank: Advanced Member

Groups: Registered
Joined: 8/11/2018(UTC)
Posts: 60
Man
Austria

Thanks: 33 times
Was thanked: 6 time(s) in 6 post(s)
TURN-support heavenly depends on UDP hole punching on both (NAT) sites of the connection, which depends on the firewall used on both ends. So maybe the remote station had the issue?

You could test UDP hole punching via http://nattest.net.in.tum.de/test.php

Thomas

hostmaster@passepartout.net  
#11 Posted : Thursday, September 27, 2018 12:39:00 PM(UTC)
hostmaster@passepartout.net

Rank: Newbie

Groups: Registered
Joined: 9/22/2018(UTC)
Posts: 5
Man
Location: Rimini

Hi everybody

Finally I got all my stuff back in my lab and I've been able to troubleshoot the problem.
I've been able to reproduce the same error even if I've not 100% understood it completely

I turned the main vMix hosting the calls and 2 of the remote vMixCall

I just describe the steps

1. Launched the first with same preset that had no problem and was continuing to work perfectly
2. Launched the second with same preset that video delay problem and was continuing to have delay problem
3. Checked vMix Call Buffer on the second and it was increasing over and over
4. Copied the preset from the first to the second and the problem disappeared
5. Launched again the second as in step 2 but changing the vMix Call pointing to another one Hosted on the main vMix. Again video delay
6. Launched again the second as in step 4 but changing the vMix Call pointing to the same having problem. No video delay

At this point I got that the problem was definitely with the preset so I reproduced the error with the original preset and I started to close one by one the input except for the vMix Call.
Right after closing the first webcam the vMix call video resyncronized.
I've started over with the same preset to check what was wrong with the input and I found out that the Sharpen option was ticked.
Unchecking that made the vMix buffer to go down around 100ms solving the delay

What it's puzzling me is that then I ticked the same flag on the first so to reproduce the error on that, but in this case nothing happened...

At the end it was just a misconfiguration, even if I prepared all the presets starting from the same file

Meanwhile looking after all of this I stumbled upon test.webrtc.org that I'm sure many already know but it proved to be a good tool for a first check on remote locations

Thanks everybody for helping

Regards
Marco



Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.