GPU Overloaded at 48% - Overload warnings, but the GPU doesn't seem to be working that hard Options  Watch this topic  Print this topic » Normal Threaded

Previous Topic Next Topic

Geoff B		#1 Posted : Wednesday, July 15, 2020 9:07:57 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		I took this screenshot... I'm getting "GPU Overloaded" warnings, but the task manager makes it look like the GPU isn't anywhere near its limit. This is running on AWS EC2 (g4dn.16xlarge). There are calls, NDI sources (and outputs), and most everything else is a PNG over a looping mp4.
		WWW
User Profile View All Posts by User View Thanks

	Wanna join the discussion?! Login to your vMix Forums forum account, or Register a new forum account.

IceStream		#2 Posted : Wednesday, July 15, 2020 9:26:03 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 3/7/2012(UTC) Posts: 2,636 Location: Canada Thanks: 33 times Was thanked: 506 time(s) in 475 post(s)		@ Geoff B Please review this article in the vMix Knowledge Base: https://www.vmix.com/kno...n-in-the-vmix-status-bar Ice
		WWW
User Profile View All Posts by User View Thanks

Geoff B		#3 Posted : Wednesday, July 15, 2020 9:31:37 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		Originally Posted by: IceStream @ Geoff B Please review this article in the vMix Knowledge Base: https://www.vmix.com/kno...n-in-the-vmix-status-bar Ice Ice, Thanks for your reply. It's not GPU memory that I'm worried about. That's sitting at 56%, and not a concern at all. The GPU performance, on the other hand, seems to be an issue. According to Task Manager, we're only using ~48% of the GPU... but I'm getting "GPU Overloaded" warnings.
		WWW
User Profile View All Posts by User View Thanks

mjgraves		#4 Posted : Wednesday, July 15, 2020 10:48:01 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 7/1/2015(UTC) Posts: 1,151 Location: Houston TX Thanks: 319 times Was thanked: 263 time(s) in 233 post(s)		Like many video processing applications, vMix make heavy use of 2D functions of the GPU. Not so much use of the 3D facilities. As such, that very general GPU usage indication may not be meaningful or insightful.
		WWW BLOG
1 user thanked mjgraves for this useful post.		Geoff B on 7/16/2020(UTC)
User Profile View All Posts by User View Thanks

simond83		#5 Posted : Wednesday, July 15, 2020 12:29:08 PM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/25/2020(UTC) Posts: 31 Was thanked: 1 time(s) in 1 post(s)		Is the GPU shared via nVidia Grid? Or is it 100% dedicated to your instance?
User Profile View All Posts by User View Thanks

Geoff B		#6 Posted : Wednesday, July 15, 2020 12:41:50 PM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		Originally Posted by: simond83 Is the GPU shared via nVidia Grid? Or is it 100% dedicated to your instance? Yes, this is a dedicated instance with its own Tesla T4.
		WWW
User Profile View All Posts by User View Thanks

Geoff B		#7 Posted : Thursday, July 16, 2020 2:27:47 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		Originally Posted by: mjgraves Like many video processing applications, vMix make heavy use of 2D functions of the GPU. Not so much use of the 3D facilities. As such, that very general GPU usage indication may not be meaningful or insightful. Thanks for your knowledge. So is there no accurate way to measure how hard a GPU is working when running vMix?
		WWW
User Profile View All Posts by User View Thanks

mjgraves		#8 Posted : Thursday, July 16, 2020 3:20:12 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 7/1/2015(UTC) Posts: 1,151 Location: Houston TX Thanks: 319 times Was thanked: 263 time(s) in 233 post(s)		Originally Posted by: Geoff B Thanks for your knowledge. So is there no accurate way to measure how hard a GPU is working when running vMix? I would try GPU-Z. It's likely more robust than the Windows built-in stuff.
		WWW BLOG
User Profile View All Posts by User View Thanks

Geoff B		#9 Posted : Thursday, July 16, 2020 3:27:43 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		Originally Posted by: mjgraves I would try GPU-Z. It's likely more robust than the Windows built-in stuff. Thanks! I'll give it a shot.
		WWW
User Profile View All Posts by User View Thanks

mjgraves		#10 Posted : Friday, July 17, 2020 4:57:10 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 7/1/2015(UTC) Posts: 1,151 Location: Houston TX Thanks: 319 times Was thanked: 263 time(s) in 233 post(s)		Originally Posted by: Geoff B Yes, this is a dedicated instance with its own Tesla T4. The T4 is a special processor designed for cloud/AI. https://www.nvidia.com/en-us/data-center/tesla-t4/ While it has a decent number of 'cuda cores, it's not clear how it performs in the traditional GPU role. It's not really built for that. What they describe with respect to video performance is high-volume decoding, because stream analysis is part of AI training.
		WWW BLOG
User Profile View All Posts by User View Thanks

Geoff B		#11 Posted : Friday, July 17, 2020 5:09:43 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		Originally Posted by: mjgraves While it has a decent number of 'cuda cores, it's not clear how it performs in the traditional GPU role. It's not really built for that. What they describe with respect to video performance is high-volume decoding, because stream analysis is part of AI training. Totally agreed. If I had a better option, I'd use it.
		WWW
User Profile View All Posts by User View Thanks

simond83		#12 Posted : Sunday, July 19, 2020 6:06:30 PM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/25/2020(UTC) Posts: 31 Was thanked: 1 time(s) in 1 post(s)		Originally Posted by: Geoff B Originally Posted by: mjgraves Totally agreed. If I had a better option, I'd use it. Have you tried attaching a second GPU to the machine? Which cloud are you on? Google has a few options; https://cloud.google.com...gpu-virtual-workstations
User Profile View All Posts by User View Thanks

grantcoll		#13 Posted : Sunday, July 19, 2020 9:05:21 PM(UTC)
Rank: Advanced Member Groups: Registered Joined: 10/7/2017(UTC) Posts: 94 Location: Hamilton Thanks: 21 times Was thanked: 10 time(s) in 9 post(s)		I wonder if the CPU is the cause, or a consequence. The Rendering is way too high. It should be below 20ms. Perhaps there is some other problem, making the Rendering bottle neck and the GPU overload could be a result. I notice that you are sending on ethernet at about 160Mbps and receiving at 697Mbps. That is a huge amount of data transfer speed. Perhaps that is some of the trouble.
User Profile View All Posts by User View Thanks

Geoff B		#14 Posted : Sunday, July 19, 2020 10:31:11 PM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		Originally Posted by: simond83 Have you tried attaching a second GPU to the machine? Which cloud are you on? Google has a few options; https://cloud.google.com...gpu-virtual-workstations We have tried multiple-GPU instances. Nothing we did improved performance. We tried attatching the virtual display -- which feeds TeamViewer or Teradici -- to one of the GPUs, and assigning vMix to the other. While it looked like the workload was being divided between the two GPUs, we still had the performance issues. We haven't tried the Google Cloud yet, but we plan on trying this out next month.
		WWW
User Profile View All Posts by User View Thanks

Geoff B		#15 Posted : Sunday, July 19, 2020 10:33:33 PM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		Originally Posted by: grantcoll I wonder if the CPU is the cause, or a consequence. The Rendering is way too high. It should be below 20ms. Perhaps there is some other problem, making the Rendering bottle neck and the GPU overload could be a result. I notice that you are sending on ethernet at about 160Mbps and receiving at 697Mbps. That is a huge amount of data transfer speed. Perhaps that is some of the trouble. The high data is due to NDI. Each stream is about 100Mbps. These machines are capable of 50Gbps transfer, so it seems unlikely to me that bandwidth is the issue.
		WWW
User Profile View All Posts by User View Thanks

simond83		#16 Posted : Monday, July 20, 2020 11:46:57 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/25/2020(UTC) Posts: 31 Was thanked: 1 time(s) in 1 post(s)		What's your machine spec? CPU/Ram etc? SSD or HDD?
User Profile View All Posts by User View Thanks

Geoff B		#17 Posted : Sunday, July 26, 2020 12:25:41 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 5/10/2020(UTC) Posts: 104 Location: Sacramento, California Thanks: 13 times Was thanked: 5 time(s) in 5 post(s)		Originally Posted by: simond83 What's your machine spec? CPU/Ram etc? SSD or HDD? This is a virtual machine (one of many) running on AWS EC2. It's a g4dn.16xlarge: 64 vCPUs 256 GB RAM 1x300GB NVMe for system 1x900GB SSD 50GBps Network Throughput
		WWW
User Profile View All Posts by User View Thanks

Users browsing this topic

Guest (3)