logo

Live Production Software Forums


Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
Geoff B  
#1 Posted : Wednesday, July 15, 2020 9:07:57 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
I took this screenshot... I'm getting "GPU Overloaded" warnings, but the task manager makes it look like the GPU isn't anywhere near its limit. This is running on AWS EC2 (g4dn.16xlarge). There are calls, NDI sources (and outputs), and most everything else is a PNG over a looping mp4.
IceStream  
#2 Posted : Wednesday, July 15, 2020 9:26:03 AM(UTC)
IceStream

Rank: Advanced Member

Groups: Registered
Joined: 3/7/2012(UTC)
Posts: 2,600
Man
Location: Canada

Thanks: 33 times
Was thanked: 501 time(s) in 470 post(s)
@ Geoff B

Please review this article in the vMix Knowledge Base:
https://www.vmix.com/kno...n-in-the-vmix-status-bar


Ice
Geoff B  
#3 Posted : Wednesday, July 15, 2020 9:31:37 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
Originally Posted by: IceStream Go to Quoted Post
@ Geoff B

Please review this article in the vMix Knowledge Base:
https://www.vmix.com/kno...n-in-the-vmix-status-bar


Ice


Ice,

Thanks for your reply. It's not GPU memory that I'm worried about. That's sitting at 56%, and not a concern at all.

The GPU performance, on the other hand, seems to be an issue. According to Task Manager, we're only using ~48% of the GPU... but I'm getting "GPU Overloaded" warnings.

mjgraves  
#4 Posted : Wednesday, July 15, 2020 10:48:01 AM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
Like many video processing applications, vMix make heavy use of 2D functions of the GPU. Not so much use of the 3D facilities. As such, that very general GPU usage indication may not be meaningful or insightful.
thanks 1 user thanked mjgraves for this useful post.
Geoff B on 7/16/2020(UTC)
simond83  
#5 Posted : Wednesday, July 15, 2020 12:29:08 PM(UTC)
simond83

Rank: Advanced Member

Groups: Registered
Joined: 5/25/2020(UTC)
Posts: 31

Was thanked: 1 time(s) in 1 post(s)
Is the GPU shared via nVidia Grid? Or is it 100% dedicated to your instance?
Geoff B  
#6 Posted : Wednesday, July 15, 2020 12:41:50 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
Originally Posted by: simond83 Go to Quoted Post
Is the GPU shared via nVidia Grid? Or is it 100% dedicated to your instance?


Yes, this is a dedicated instance with its own Tesla T4.

Geoff B  
#7 Posted : Thursday, July 16, 2020 2:27:47 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
Originally Posted by: mjgraves Go to Quoted Post
Like many video processing applications, vMix make heavy use of 2D functions of the GPU. Not so much use of the 3D facilities. As such, that very general GPU usage indication may not be meaningful or insightful.


Thanks for your knowledge. So is there no accurate way to measure how hard a GPU is working when running vMix?

mjgraves  
#8 Posted : Thursday, July 16, 2020 3:20:12 AM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
Originally Posted by: Geoff B Go to Quoted Post

Thanks for your knowledge. So is there no accurate way to measure how hard a GPU is working when running vMix?


I would try GPU-Z. It's likely more robust than the Windows built-in stuff.

Geoff B  
#9 Posted : Thursday, July 16, 2020 3:27:43 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
Originally Posted by: mjgraves Go to Quoted Post
I would try GPU-Z. It's likely more robust than the Windows built-in stuff.


Thanks! I'll give it a shot.

mjgraves  
#10 Posted : Friday, July 17, 2020 4:57:10 AM(UTC)
mjgraves

Rank: Advanced Member

Groups: Registered
Joined: 7/1/2015(UTC)
Posts: 1,150
Man
United States
Location: Houston TX

Thanks: 319 times
Was thanked: 263 time(s) in 233 post(s)
Originally Posted by: Geoff B Go to Quoted Post

Yes, this is a dedicated instance with its own Tesla T4.


The T4 is a special processor designed for cloud/AI.

https://www.nvidia.com/en-us/data-center/tesla-t4/

While it has a decent number of 'cuda cores, it's not clear how it performs in the traditional GPU role. It's not really built for that.

What they describe with respect to video performance is high-volume decoding, because stream analysis is part of AI training.
Geoff B  
#11 Posted : Friday, July 17, 2020 5:09:43 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
Originally Posted by: mjgraves Go to Quoted Post
While it has a decent number of 'cuda cores, it's not clear how it performs in the traditional GPU role. It's not really built for that.

What they describe with respect to video performance is high-volume decoding, because stream analysis is part of AI training.


Totally agreed. If I had a better option, I'd use it.

simond83  
#12 Posted : Sunday, July 19, 2020 6:06:30 PM(UTC)
simond83

Rank: Advanced Member

Groups: Registered
Joined: 5/25/2020(UTC)
Posts: 31

Was thanked: 1 time(s) in 1 post(s)
Originally Posted by: Geoff B Go to Quoted Post
Originally Posted by: mjgraves Go to Quoted Post
Totally agreed. If I had a better option, I'd use it.


Have you tried attaching a second GPU to the machine? Which cloud are you on?

Google has a few options;
https://cloud.google.com...gpu-virtual-workstations
grantcoll  
#13 Posted : Sunday, July 19, 2020 9:05:21 PM(UTC)
grantcoll

Rank: Advanced Member

Groups: Registered
Joined: 10/7/2017(UTC)
Posts: 94
Location: Hamilton

Thanks: 21 times
Was thanked: 10 time(s) in 9 post(s)
I wonder if the CPU is the cause, or a consequence. The Rendering is way too high. It should be below 20ms. Perhaps there is some other problem, making the Rendering bottle neck and the GPU overload could be a result. I notice that you are sending on ethernet at about 160Mbps and receiving at 697Mbps. That is a huge amount of data transfer speed. Perhaps that is some of the trouble.

Geoff B  
#14 Posted : Sunday, July 19, 2020 10:31:11 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
Originally Posted by: simond83 Go to Quoted Post
Have you tried attaching a second GPU to the machine? Which cloud are you on?

Google has a few options;
https://cloud.google.com...gpu-virtual-workstations


We have tried multiple-GPU instances. Nothing we did improved performance. We tried attatching the virtual display -- which feeds TeamViewer or Teradici -- to one of the GPUs, and assigning vMix to the other. While it looked like the workload was being divided between the two GPUs, we still had the performance issues.

We haven't tried the Google Cloud yet, but we plan on trying this out next month.

Geoff B  
#15 Posted : Sunday, July 19, 2020 10:33:33 PM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
Originally Posted by: grantcoll Go to Quoted Post
I wonder if the CPU is the cause, or a consequence. The Rendering is way too high. It should be below 20ms. Perhaps there is some other problem, making the Rendering bottle neck and the GPU overload could be a result. I notice that you are sending on ethernet at about 160Mbps and receiving at 697Mbps. That is a huge amount of data transfer speed. Perhaps that is some of the trouble.



The high data is due to NDI. Each stream is about 100Mbps. These machines are capable of 50Gbps transfer, so it seems unlikely to me that bandwidth is the issue.
simond83  
#16 Posted : Monday, July 20, 2020 11:46:57 AM(UTC)
simond83

Rank: Advanced Member

Groups: Registered
Joined: 5/25/2020(UTC)
Posts: 31

Was thanked: 1 time(s) in 1 post(s)
What's your machine spec? CPU/Ram etc?

SSD or HDD?
Geoff B  
#17 Posted : Sunday, July 26, 2020 12:25:41 AM(UTC)
Geoff B

Rank: Advanced Member

Groups: Registered
Joined: 5/10/2020(UTC)
Posts: 104
Man
United States
Location: Sacramento, California

Thanks: 13 times
Was thanked: 5 time(s) in 5 post(s)
Originally Posted by: simond83 Go to Quoted Post
What's your machine spec? CPU/Ram etc?

SSD or HDD?


This is a virtual machine (one of many) running on AWS EC2. It's a g4dn.16xlarge:

64 vCPUs
256 GB RAM
1x300GB NVMe for system
1x900GB SSD
50GBps Network Throughput


Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.