logo

Live Production Software Forums


Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
bdaver  
#1 Posted : Saturday, April 17, 2021 8:57:12 PM(UTC)
bdaver

Rank: Member

Groups: Registered
Joined: 4/17/2021(UTC)
Posts: 10
United Kingdom
Location: London

Was thanked: 1 time(s) in 1 post(s)
NVIDIA Broadcasting and Maxine SDK offers many features we can enjoy
- Audio Noise cancellation ( can be giant improvement over noise gate/eq etc )
- Background removal ( other people also requested )
- Reframing the input using head tracking ( auto input zoom, pan )
- Eye contact / Gaze correction ( this can be huge. Most vMix Call attendees are not pros. They dont look at their webcams at all and they dont have teleprompters )
- Speech recognition and translation ( can be nice with GT/Title system integration )

Codec based solutions (ultra low bandwidth) included in maxine sdk might be impossible to implement in current state of WebRTC and codec adoption.

Up if you would like it !
Babbit  
#2 Posted : Sunday, April 18, 2021 3:04:55 AM(UTC)
Babbit

Rank: Advanced Member

Groups: Registered
Joined: 10/26/2020(UTC)
Posts: 76
United States
Location: Maryland

Thanks: 2 times
Was thanked: 26 time(s) in 20 post(s)
It's a least worth a gander, but I'm concerned that it would eat up too much GPU for the rest of vMix to run well.

You have to draw the line somewhere, and unfortunately people tend to think everything offered means everything can be run simultaneously. I could see someone trying this in 4K while running instant replay, and expressing misdirected frustration when it all comes crashing down.
thanks 1 user thanked Babbit for this useful post.
doggy on 4/18/2021(UTC)
bdaver  
#3 Posted : Tuesday, April 20, 2021 4:49:56 AM(UTC)
bdaver

Rank: Member

Groups: Registered
Joined: 4/17/2021(UTC)
Posts: 10
United Kingdom
Location: London

Was thanked: 1 time(s) in 1 post(s)
Of course it may eat up a lot regarding on the GPU you're running.
RTX noise cancellation has 2-3% impact on recent cards but I can't say even if it's backward compatible with older cards.
There are ways to optimize. Instead running noise cancellation per call, you can run it on a audio bus where every call mixed down.

🤞 fingers crossed 🤞
bdaver  
#4 Posted : Thursday, May 6, 2021 1:15:53 AM(UTC)
bdaver

Rank: Member

Groups: Registered
Joined: 4/17/2021(UTC)
Posts: 10
United Kingdom
Location: London

Was thanked: 1 time(s) in 1 post(s)
https://developer.nvidia.com/maxine-getting-started

I received a mail regarding SDK availability about an hour ago.

buffos  
#5 Posted : Sunday, May 9, 2021 8:04:09 PM(UTC)
buffos

Rank: Member

Groups: Registered
Joined: 5/9/2021(UTC)
Posts: 15
Greece
Location: Thessaloniki

Thanks: 2 times
Originally Posted by: bdaver Go to Quoted Post
https://developer.nvidia.com/maxine-getting-started

I received a mail regarding SDK availability about an hour ago.



Isn't it possible to use the Nvidia broadcast app on your camera and then import that camera in Vmix?


Babbit  
#6 Posted : Monday, May 10, 2021 1:46:06 PM(UTC)
Babbit

Rank: Advanced Member

Groups: Registered
Joined: 10/26/2020(UTC)
Posts: 76
United States
Location: Maryland

Thanks: 2 times
Was thanked: 26 time(s) in 20 post(s)
I ran the Broadcast App noise cancelling for the first time tonight. Except for one well-formed chair squeak, it filtered out noise like a champ. Apologies to those who enjoy hearing needy cats.

One interesting thing to note is that ASIO isn't supported. I had to feed the traditional UMC404HD 1-2 input into the program. Everything else was smooth sailing though.
bdaver  
#7 Posted : Monday, May 10, 2021 8:27:40 PM(UTC)
bdaver

Rank: Member

Groups: Registered
Joined: 4/17/2021(UTC)
Posts: 10
United Kingdom
Location: London

Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: buffos Go to Quoted Post

Isn't it possible to use the Nvidia broadcast app on your camera and then import that camera in Vmix?


Of course local usage of some of the SDK features are available today with the broadcast app but with such SDK integrated into vMix, all vMix call attendees can benefit from it without anything installed on their computer. And I believe there are more features to benefit from this SDK.

  • Virtual background - Beta
  • Super resolution (up to 4X scaling factor) - Beta
  • Artifact reduction - Beta
  • Video noise removal - Beta
  • Noise removal - V1.0
  • Room echo removal - Beta
  • Eye contact - Coming Soon
  • Face tracking - Windows in Beta, Linux in Alpha
thanks 1 user thanked bdaver for this useful post.
contact@german-pictures.de on 4/16/2023(UTC)
mgrayeb  
#8 Posted : Wednesday, June 2, 2021 1:49:46 PM(UTC)
mgrayeb

Rank: Advanced Member

Groups: Registered
Joined: 3/3/2018(UTC)
Posts: 184

Thanks: 86 times
Was thanked: 7 time(s) in 7 post(s)
Originally Posted by: bdaver Go to Quoted Post
Originally Posted by: buffos Go to Quoted Post

Isn't it possible to use the Nvidia broadcast app on your camera and then import that camera in Vmix?


Of course local usage of some of the SDK features are available today with the broadcast app but with such SDK integrated into vMix, all vMix call attendees can benefit from it without anything installed on their computer. And I believe there are more features to benefit from this SDK.

  • Virtual background - Beta
  • Super resolution (up to 4X scaling factor) - Beta
  • Artifact reduction - Beta
  • Video noise removal - Beta
  • Noise removal - V1.0
  • Room echo removal - Beta
  • Eye contact - Coming Soon
  • Face tracking - Windows in Beta, Linux in Alpha


I have an nVidia GTX-2060 graphics card. Are these features available with that card and if yes, how do you enable them in vMix? I'm hoping to reduce room noise, etc. from multiple mics I'm bringing into vMix via a Behringer 404 UMCHD audio interface (up to 6 Rode Wireless Go II mics). Thank you.
mtone  
#9 Posted : Wednesday, June 2, 2021 3:18:16 PM(UTC)
mtone

Rank: Advanced Member

Groups: Registered
Joined: 8/9/2020(UTC)
Posts: 156

Thanks: 3 times
Was thanked: 20 time(s) in 19 post(s)
it would be really good to get object tracking applied to PTZ.. drones and gimbals let you draw a box around your subject and then track it, i dont think it is that processor intensive ? this sort of automation would be helpful when running productions with only one person or limited crew...
bdaver  
#10 Posted : Saturday, June 5, 2021 6:04:22 AM(UTC)
bdaver

Rank: Member

Groups: Registered
Joined: 4/17/2021(UTC)
Posts: 10
United Kingdom
Location: London

Was thanked: 1 time(s) in 1 post(s)
To enable nVidia Broadcast in your vMix workflow, try the following:

For video effects
1. Expose your vMix A/V source as NDI (in my case, I am using it with vMix call)
2. Use Webcam Input from NDI tools to convert NDI to virtual webcam (vMix external is not detected by NVIDIA Broadcast)
3. Open NVIDIA Broadcast and select "Newtek NDI Video" as your camera source
4. Add effects (background replacement, Auto frame etc.)
5. in vMix, add Camera input and select NVIDIA Broadcast

For audio effects using Newtek NDI Audio produces jittery results. I found the following workaround;
1. Install any virtual soundcard driver
2. in vMix settings > Audio Outputs , select your virtual soundcard in any of the busses (lets say A)
3. Send the audio of the input source you want to process to Audio Bus A
4. Select your virtual soundcard in NVIDIA Broadcast Microphone settings
5. Add effects (Noise removal, room echo removal)
6. Same as the video effects step 5 above, you can bundle the NVIDIA broadcast audio along with the video while adding source back into vMix.

Dirty solution but it works.

note : Background removal filter works! Resulting webcam image have an alpha channel. So you don't have to key in vmix.
bdaver  
#11 Posted : Saturday, June 5, 2021 6:11:08 AM(UTC)
bdaver

Rank: Member

Groups: Registered
Joined: 4/17/2021(UTC)
Posts: 10
United Kingdom
Location: London

Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: mgrayeb Go to Quoted Post

I have an nVidia GTX-2060 graphics card. Are these features available with that card and if yes, how do you enable them in vMix? I'm hoping to reduce room noise, etc. from multiple mics I'm bringing into vMix via a Behringer 404 UMCHD audio interface (up to 6 Rode Wireless Go II mics). Thank you.


You should be able to use NVIDIA Broadcast with RTX-2060. Make sure you have the latest drivers

If you can select Behringer 404 in NVIDIA Broadcast Microphone source, just add NVIDIA Broadcast as Audio Input in vMix.
If not, follow my previous message ^
Users browsing this topic
Guest (3)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.