logo

Live Production Software Forums


Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
bgadient  
#1 Posted : Thursday, January 13, 2022 4:53:19 AM(UTC)
bgadient

Rank: Newbie

Groups: Registered
Joined: 1/13/2022(UTC)
Posts: 2
United States
Location: Minnesota

Was thanked: 2 time(s) in 1 post(s)
For quite some time we have had issues with Vmix where about once a stream, the internet upload would be cut to a speed that would either cripple the stream or would stop the stream. It would only last for about a minute and then go back to normal. We've had no issues after our IT guy updated things. (see below) Anyone else have the same issue with the outdated FFMPEG and Youtube?

We have a 1gig upload and a high end pc. Here are the main specs of the pc...

Precision 5820 Tower
Precision 5820 Tower XCTO Base
Processor
Intel Core i9-10900X (3.7GHz, 4.7GHz Turbo, 10C, 19.25MB Cache, HT, (165W)), DDR4-2933 Non-ECC
Operating System
Windows 11 Pro, English, French, Spanish
Chassis Options
Precision 5820 Tower Core X 950W PCIe FlexBay Chassis CL FMX
Video Card
Nvidia Quadro RTX4000, 8GB, 3DP, VirtualLink (XX20T)
Memory
32GB, DDR4 UDIMM non-ECC memory

"I think I have a solution for your issue. I have replaced the FFMPEG2 binary in vMix with one I built which so far has resolved the issue. To use it select FFMPEG2 when setting up streaming, the FFMPEG option will use the default version included with vMix. The FFMPEG I've included a more complete trip down the rabbit hole below if you are interested below.

image.png (107kb) downloaded 3 time(s).

Here is an overview of the steps I took to get to this point:
1. I set up vMix to stream to YouTube's primary and secondary servers at the same time. This is mentioned as a "fix" on several of the vMix forums along with using the Flash Media Encoder from Adobe. During a ~18 hour run both the primary and secondary streams restarted due to buffer errors around 8 times but neither stream restarted within an hour or two of the other stream. Any network congestion or packet loss issues between the computer and the point in Google's network where their paths split should show up on both streams at the same time. This pointed to either the issue being on the computer or a remote possibility of something going on randomly in Google's network.

2. I completed a full network packet capture on the computer during a series of 4-5 test streams and observed traffic up until the point of failure. An analysis of the TCP traffic flows didn't show any packet loss or loss of ACKs during the time leading up to the failure. In all of the cases Google's server sent an ACK and then there was ~30s of silence from the vMix computer. When the connection timed out Google's server would send a FIN packet to close the connection after which the vMix computer would try to send a bunch of traffic before stopping. This pointed to some sort of race condition where FFMPEG on the vMIx computer and YouTube's server are both waiting on eachother for data.

3. I then examined FFMPEG on the vMix computer. vMix has ffmpeg.exe and ffmpeg2.exe which are stored in the streaming directory of the vMix application in Program File (x86). ffmpeg.exe appears to have been built from the FFMPEG 2.8 master branch in 2014. ffmpeg2.exe appears to have been built from the FFMPEG 3.3 master branch in 2017. I performed a stream with FFMPEG2 but it exhibited the same issue as FFMPEG1. I downloaded several past versions of vMix hoping to see a rebuild in recent history that is causing this error but file checksums are the same for vMix versions going back to 2015/2017 indicating that nothing has changed. At this point I suspected the issue could be something in the local network stack or a bug in the old version of FFMPEG. I then used OBS on the vMix computer, which uses modern FFMPEG libraries for streaming, to complete a 16 hour stream to YouTube without any dropped frames or quality issues which seemed to confirm that even if there is some issue in the network stack that the modern FFMPEG libraries are able to handle it gracefully.

4. I looked online for more modern FFMPEG binaries that incorporated both the NVIDIA decoding plugin and Windows specific DirectShow support but due to proprietary licensing issues they are non-existent from reputable sources. Given that, I downloaded the latest NVIDIA SDK, MinGW-W64 build environment, and the latest stable FFMPEG version and produced a modern 64-bit FFMPEG 4.4 binary to be a drop in replacement for the FFMPEG2 binary. The original binary was built using the MinGW-W64 gcc compilers so I opted to do the same rather than try using Microsoft's Visual C compilers which they do for the GUI components of vMix but not the streaming command line version. I did a couple of short tests with the new binary before moving on to longer tests. So far I have completed a 24.5 hour stream over the weekend and a 15 hour stream from late last night up until the elementary concert started without a crash or the network hang issue occurring. I can't declare victory yet, but so far the results are promising. If I were working for vMix I'd work further to determine the exact issue so that a patch could be backported, but for now if it works or masks the issue so I'll leave it at that. My binary is 32 vs 64 bit, uses a much newer NVIDIA SDK, and has different GCC compilers and libraries so the root cause could be in a whole bunch of different places.

It would be nice if Vmix would keep their FFMPEG versions up to date with the latest maintenance branch. Version 2.8 and 3.3 are both still getting critical fixes and patches but it doesn't appear like they've ever really updated things. I didn't try building the latest 2.8 or 3.3 versions to see if a backported patch is already in those. If they'd add FFMPEG 4 they would have a version with HLS support which Google has added to YouTube to increase reliability and add HDR support. "
thanks 2 users thanked bgadient for this useful post.
markleman on 3/14/2022(UTC), pba on 3/19/2022(UTC)
markleman  
#2 Posted : Monday, March 14, 2022 3:22:21 AM(UTC)
markleman

Rank: Member

Groups: Registered
Joined: 8/7/2015(UTC)
Posts: 28

Thanks: 9 times
Was thanked: 2 time(s) in 2 post(s)
Thanks for this write up, I too wondered why the version of FFMPEG in vmix us so old. I thought it could be due to the 'fall out' between Newtek/NDI and FFMPEG (https://trac.ffmpeg.org/ticket/7589), but then I noticed that even the newer version of FFMPEG included in vmix predates all this.

This thread https://forums.vmix.com/posts/t20431-FFMPEG-Update does say they were going to update to "FFMPEG 4.1.3 in vMix 23" but I guess that never happened.

Any notes from your IT guy recompiling FFMPEG on windows? I found this:
As I would really like like a newer version of FFMPEG with NDI included it might be time to follow those instructions, along with https://framagit.org/tytan652/ffmpeg-ndi-patch

Regards,
Mark Leman
NiBTour  
#3 Posted : Wednesday, March 16, 2022 5:46:50 PM(UTC)
NiBTour

Rank: Advanced Member

Groups: Registered
Joined: 3/28/2016(UTC)
Posts: 155
Man
United States
Location: SACRAMENTO

Thanks: 5 times
Was thanked: 18 time(s) in 17 post(s)
Any chance you could put up a download link for the compiled version?

I'm pretty sure I read someplace that having such an old ffmpeg version was simply due to there was no need to update it. It was solid so why mess with it. But as your guy pointed out other features that could benefit us would make it ideal if it was more like OBS where we could just pass own our own command lines or use our own builds where the drop-down would be our profile so we could use more features as well as even different encoders. I know it can be done now but make it more like OBS where it's right there in the advanced features of the GUI.

nice job

Originally Posted by: bgadient Go to Quoted Post
For quite some time we have had issues with Vmix where about once a stream, the internet upload would be cut to a speed that would either cripple the stream or would stop the stream. It would only last for about a minute and then go back to normal. We've had no issues after our IT guy updated things. (see below) Anyone else have the same issue with the outdated FFMPEG and Youtube?

We have a 1gig upload and a high end pc. Here are the main specs of the pc...

Precision 5820 Tower
Precision 5820 Tower XCTO Base
Processor
Intel Core i9-10900X (3.7GHz, 4.7GHz Turbo, 10C, 19.25MB Cache, HT, (165W)), DDR4-2933 Non-ECC
Operating System
Windows 11 Pro, English, French, Spanish
Chassis Options
Precision 5820 Tower Core X 950W PCIe FlexBay Chassis CL FMX
Video Card
Nvidia Quadro RTX4000, 8GB, 3DP, VirtualLink (XX20T)
Memory
32GB, DDR4 UDIMM non-ECC memory

"I think I have a solution for your issue. I have replaced the FFMPEG2 binary in vMix with one I built which so far has resolved the issue. To use it select FFMPEG2 when setting up streaming, the FFMPEG option will use the default version included with vMix. The FFMPEG I've included a more complete trip down the rabbit hole below if you are interested below.

image.png (107kb) downloaded 3 time(s).

Here is an overview of the steps I took to get to this point:
1. I set up vMix to stream to YouTube's primary and secondary servers at the same time. This is mentioned as a "fix" on several of the vMix forums along with using the Flash Media Encoder from Adobe. During a ~18 hour run both the primary and secondary streams restarted due to buffer errors around 8 times but neither stream restarted within an hour or two of the other stream. Any network congestion or packet loss issues between the computer and the point in Google's network where their paths split should show up on both streams at the same time. This pointed to either the issue being on the computer or a remote possibility of something going on randomly in Google's network.

2. I completed a full network packet capture on the computer during a series of 4-5 test streams and observed traffic up until the point of failure. An analysis of the TCP traffic flows didn't show any packet loss or loss of ACKs during the time leading up to the failure. In all of the cases Google's server sent an ACK and then there was ~30s of silence from the vMix computer. When the connection timed out Google's server would send a FIN packet to close the connection after which the vMix computer would try to send a bunch of traffic before stopping. This pointed to some sort of race condition where FFMPEG on the vMIx computer and YouTube's server are both waiting on eachother for data.

3. I then examined FFMPEG on the vMix computer. vMix has ffmpeg.exe and ffmpeg2.exe which are stored in the streaming directory of the vMix application in Program File (x86). ffmpeg.exe appears to have been built from the FFMPEG 2.8 master branch in 2014. ffmpeg2.exe appears to have been built from the FFMPEG 3.3 master branch in 2017. I performed a stream with FFMPEG2 but it exhibited the same issue as FFMPEG1. I downloaded several past versions of vMix hoping to see a rebuild in recent history that is causing this error but file checksums are the same for vMix versions going back to 2015/2017 indicating that nothing has changed. At this point I suspected the issue could be something in the local network stack or a bug in the old version of FFMPEG. I then used OBS on the vMix computer, which uses modern FFMPEG libraries for streaming, to complete a 16 hour stream to YouTube without any dropped frames or quality issues which seemed to confirm that even if there is some issue in the network stack that the modern FFMPEG libraries are able to handle it gracefully.

4. I looked online for more modern FFMPEG binaries that incorporated both the NVIDIA decoding plugin and Windows specific DirectShow support but due to proprietary licensing issues they are non-existent from reputable sources. Given that, I downloaded the latest NVIDIA SDK, MinGW-W64 build environment, and the latest stable FFMPEG version and produced a modern 64-bit FFMPEG 4.4 binary to be a drop in replacement for the FFMPEG2 binary. The original binary was built using the MinGW-W64 gcc compilers so I opted to do the same rather than try using Microsoft's Visual C compilers which they do for the GUI components of vMix but not the streaming command line version. I did a couple of short tests with the new binary before moving on to longer tests. So far I have completed a 24.5 hour stream over the weekend and a 15 hour stream from late last night up until the elementary concert started without a crash or the network hang issue occurring. I can't declare victory yet, but so far the results are promising. If I were working for vMix I'd work further to determine the exact issue so that a patch could be backported, but for now if it works or masks the issue so I'll leave it at that. My binary is 32 vs 64 bit, uses a much newer NVIDIA SDK, and has different GCC compilers and libraries so the root cause could be in a whole bunch of different places.

It would be nice if Vmix would keep their FFMPEG versions up to date with the latest maintenance branch. Version 2.8 and 3.3 are both still getting critical fixes and patches but it doesn't appear like they've ever really updated things. I didn't try building the latest 2.8 or 3.3 versions to see if a backported patch is already in those. If they'd add FFMPEG 4 they would have a version with HLS support which Google has added to YouTube to increase reliability and add HDR support. "


pba  
#4 Posted : Saturday, March 19, 2022 12:14:19 AM(UTC)
pba

Rank: Advanced Member

Groups: Registered
Joined: 10/14/2015(UTC)
Posts: 147
Location: Hungary

Thanks: 38 times
Was thanked: 24 time(s) in 20 post(s)
Thank you for this investigation! Unfortunatelly and sadly streaming from OBS/wirecast is much more stable than from VMIX. Likely it caused by the old and outdated FFMPEG, so its really needed to update. I think its worth to open a topic for this in the feature request section.

It's so strange, no any reaction from the VMIX team, just silence. :( IMHO there is a lot of really important missing features and problems, and these things are really missing from the V25. This is a disappointment for me...
bgadient  
#5 Posted : Saturday, March 19, 2022 12:37:33 AM(UTC)
bgadient

Rank: Newbie

Groups: Registered
Joined: 1/13/2022(UTC)
Posts: 2
United States
Location: Minnesota

Was thanked: 2 time(s) in 1 post(s)
I did submit a ticket to support. Here is the response.......

"This is not an issue we have seen or reported by anyone else, it seems likely an issue with that particular machine or internet connection.

Specifically, FFMPEG versions have not had much if anything change in regards to RTMP in recent releases, and older versions have been chosen for their stability.

I would suggest reinstalling vMix to ensure the correct FFMPEG versions are installed and sending us a support report as per instructions below when the issue occurs on at least two different computers and internet connections to rule out other issues.

1. Open vMix and open the preset you would normally use where the issue occurs.
2. Click the "Settings" button in the top right corner
3. Go to the "About" tab and then select the grey "Send Support Report" button
4. Email us back to let us know you have sent the Support Report"
Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.