vMix Forums
	 » 
	General
	 » 
	Feature Requests
	 » 
	NVIDIA Broadcasting and Maxine SDK integration
	 
	
        
            
            
    | 
	Rank: Member
 Groups: Registered
 Joined: 4/17/2021(UTC) Posts: 11  Location: LondonWas thanked: 1 time(s) in 1 post(s)
 | 
            
		      
                NVIDIA Broadcasting and Maxine SDK offers many features we can enjoy- Audio Noise cancellation ( can be giant improvement over noise gate/eq etc )
 - Background removal ( other people also requested )
 - Reframing the input using head tracking ( auto input zoom, pan )
 - Eye contact / Gaze correction ( this can be huge. Most vMix Call attendees are not pros. They dont look at their webcams at all and they dont have teleprompters )
 - Speech recognition and translation ( can be nice with GT/Title system integration )
 
 Codec based solutions (ultra low bandwidth) included in maxine sdk might be impossible to implement in current state of WebRTC and codec adoption.
 
 Up if you would like it !
 | 
    | 
             | 
            
         | 
    |  | 
        
        
        
            
        
            
            
    | 
	Rank: Advanced Member
 Groups: Registered
 Joined: 10/26/2020(UTC) Posts: 76  Location: MarylandThanks: 2 timesWas thanked: 26 time(s) in 20 post(s)
 
 | 
            
		      
                It's a least worth a gander, but I'm concerned that it would eat up too much GPU for the rest of vMix to run well.
 You have to draw the line somewhere, and unfortunately people tend to think everything offered means everything can be run simultaneously. I could see someone trying this in 4K while running instant replay, and expressing misdirected frustration when it all comes crashing down.
 | 
    | 
              1 user thanked Babbit for this useful post. |  | 
    |  | 
        
        
        
    
        
            
            
    | 
	Rank: Member
 Groups: Registered
 Joined: 4/17/2021(UTC) Posts: 11  Location: LondonWas thanked: 1 time(s) in 1 post(s)
 | 
            
		      
                Of course it may eat up a lot regarding on the GPU you're running. RTX noise cancellation has 2-3% impact on recent cards but I can't say even if it's backward compatible with older cards.
 There are ways to optimize. Instead running noise cancellation per call, you can run it on a audio bus where every call mixed down.
 
 🤞 fingers crossed 🤞
 
 | 
    | 
             | 
            
         | 
    |  | 
        
        
        
            
        
            
            
    | 
	Rank: Member
 Groups: Registered
 Joined: 4/17/2021(UTC) Posts: 11  Location: LondonWas thanked: 1 time(s) in 1 post(s)
 |  | 
    | 
             | 
            
         | 
    |  | 
        
        
        
    
        
            
            
    | 
	Rank: Member
 Groups: Registered
 Joined: 5/9/2021(UTC) Posts: 15  Location: ThessalonikiThanks: 2 times
 | 
            
		      
                Originally Posted by: bdaver  Isn't it possible to use the Nvidia broadcast app on your camera and then import that camera in Vmix? | 
    | 
             | 
            
         | 
    |  | 
        
        
        
            
        
            
            
    | 
	Rank: Advanced Member
 Groups: Registered
 Joined: 10/26/2020(UTC) Posts: 76  Location: MarylandThanks: 2 timesWas thanked: 26 time(s) in 20 post(s)
 
 | 
            
		      
                I ran the Broadcast App noise cancelling for the first time tonight. Except for one well-formed chair squeak, it filtered out noise like a champ. Apologies to those who enjoy hearing needy cats.
 One interesting thing to note is that ASIO isn't supported. I had to feed the traditional UMC404HD 1-2 input into the program. Everything else was smooth sailing though.
 | 
    | 
             | 
            
         | 
    |  | 
        
        
        
    
        
            
            
    | 
	Rank: Member
 Groups: Registered
 Joined: 4/17/2021(UTC) Posts: 11  Location: LondonWas thanked: 1 time(s) in 1 post(s)
 | 
            
		      
                Originally Posted by: buffos  Isn't it possible to use the Nvidia broadcast app on your camera and then import that camera in Vmix?
 
 Of course local usage of some of the SDK features are available today with the broadcast app but with such SDK integrated into vMix, all vMix call attendees can benefit from it without anything installed on their computer. And I believe there are more features to benefit from this SDK.  Virtual background - Beta
Super resolution (up to 4X scaling factor) - Beta
Artifact reduction - Beta
Video noise removal - Beta
Noise removal - V1.0
Room echo removal - Beta
Eye contact - Coming Soon
Face tracking - Windows in Beta, Linux in Alpha
 | 
    | 
              1 user thanked bdaver for this useful post. |  | 
    |  | 
        
        
        
            
        
            
            
    | 
	Rank: Advanced Member
 Groups: Registered
Joined: 3/3/2018(UTC)
 Posts: 184
 
 Thanks: 86 timesWas thanked: 7 time(s) in 7 post(s)
 
 | 
            
		      
                Originally Posted by: bdaver  Originally Posted by: buffos  Isn't it possible to use the Nvidia broadcast app on your camera and then import that camera in Vmix?
 
 Of course local usage of some of the SDK features are available today with the broadcast app but with such SDK integrated into vMix, all vMix call attendees can benefit from it without anything installed on their computer. And I believe there are more features to benefit from this SDK.  Virtual background - Beta
Super resolution (up to 4X scaling factor) - Beta
Artifact reduction - Beta
Video noise removal - Beta
Noise removal - V1.0
Room echo removal - Beta
Eye contact - Coming Soon
Face tracking - Windows in Beta, Linux in Alpha
 I have an nVidia GTX-2060 graphics card.  Are these features available with that card and if yes, how do you enable them in vMix?  I'm hoping to reduce room noise, etc. from multiple mics I'm bringing into vMix via a Behringer 404 UMCHD audio interface (up to 6 Rode Wireless Go II mics).  Thank you. | 
    | 
             | 
            
         | 
    |  | 
        
        
        
    
        
            
            
    | 
	Rank: Advanced Member
 Groups: Registered
Joined: 8/9/2020(UTC)
 Posts: 156
 
 Thanks: 3 timesWas thanked: 20 time(s) in 19 post(s)
 
 | 
            
		      
                it would be really good to get object tracking applied to PTZ.. drones and gimbals let you draw a box around your subject and then track it, i dont think it is that processor intensive ? this sort of automation would be helpful when running productions with only one person or limited crew... | 
    | 
             | 
            
         | 
    |  | 
        
        
        
            
        
            
            
    | 
	Rank: Member
 Groups: Registered
 Joined: 4/17/2021(UTC) Posts: 11  Location: LondonWas thanked: 1 time(s) in 1 post(s)
 | 
            
		      
                To enable nVidia Broadcast in your vMix workflow, try the following:
 For video effects
 1. Expose your vMix A/V source as NDI (in my case, I am using it with vMix call)
 2. Use Webcam Input from NDI tools to convert NDI to virtual webcam (vMix external is not detected by NVIDIA Broadcast)
 3. Open NVIDIA Broadcast and select "Newtek NDI Video" as your camera source
 4. Add effects (background replacement, Auto frame etc.)
 5. in vMix, add Camera input and select NVIDIA Broadcast
 
 For audio effects using Newtek NDI Audio produces jittery results. I found the following workaround;
 1. Install any virtual soundcard driver
 2. in vMix settings > Audio Outputs , select your virtual soundcard in any of the busses (lets say A)
 3. Send the audio of the input source you want to process to Audio Bus A
 4. Select your virtual soundcard in NVIDIA Broadcast Microphone settings
 5. Add effects (Noise removal, room echo removal)
 6. Same as the video effects step 5 above, you can bundle the NVIDIA broadcast audio along with the video while adding source back into vMix.
 
 Dirty solution but it works.
 
 note : Background removal filter works! Resulting webcam image have an alpha channel. So you don't have to key in vmix.
 | 
    | 
             | 
            
         | 
    |  | 
        
        
        
    
        
            
            
    | 
	Rank: Member
 Groups: Registered
 Joined: 4/17/2021(UTC) Posts: 11  Location: LondonWas thanked: 1 time(s) in 1 post(s)
 | 
            
		      
                Originally Posted by: mgrayeb  I have an nVidia GTX-2060 graphics card.  Are these features available with that card and if yes, how do you enable them in vMix?  I'm hoping to reduce room noise, etc. from multiple mics I'm bringing into vMix via a Behringer 404 UMCHD audio interface (up to 6 Rode Wireless Go II mics).  Thank you.
 You should be able to use NVIDIA Broadcast with RTX-2060. Make sure you have the latest drivers If you can select Behringer 404 in NVIDIA Broadcast Microphone source, just add NVIDIA Broadcast as Audio Input in vMix.  If not, follow my previous message ^ | 
    | 
             | 
            
         | 
    |  | 
        
        
        
    
                           
	vMix Forums
	 » 
	General
	 » 
	Feature Requests
	 » 
	NVIDIA Broadcasting and Maxine SDK integration
	 
	
    
        Forum Jump
         
	
    
    You cannot post new topics in this forum.
	
	You cannot reply to topics in this forum.
	
	You cannot delete your posts in this forum.
	
	You cannot edit your posts in this forum.
	
	You cannot create polls in this forum.
	
	You cannot vote in polls in this forum.
	
	
    
    
        Important Information:
        The vMix Forums uses cookies. By continuing to browse this site, you are agreeing to our use of cookies.
        
        
More Details
        Close