Amazing specs. Reading it, it seems that it's maybe a source issue. You have the same with capture card and over NDI. Your computer is running low. But what if the signal that enters your computer has the stutter already.
You could try another source. DVD player or what have you and see if you can reproduce the stutter with a different source. If it happens there as well, it's not a source issue. So either computer or Youtube. If it vanishes you found the issue.

Ill record a file in camera and then put it in vmix and stream it to see if the source is the issue. Ive had this problem since 2019. Used a different computer and capture card then as well. The stutter isnt nonstop. It randomly starts happening for a few minutes then goes away.

So the issue appears to be dropped frames that arent being detected by vmix from the source. I have to close and then readd my camera source coming from my elgato 4kpro mk2 every 15-30 minutes to fix the stuttering. If i dont do this theres very obvious motion stutter. Is there a way to prevent this? Source is 59.94p in camera, elgato detects it as that, im adding it to vmix as 59.94, my vmix layout is 59.94. I can see it start stuttering in vmix with no dropped frames detected and under 10% cpu and gpu usage.

Everything seems correct in terms of connections and settings.
Have you tried swapping the HDMI cable?
It looks like the issue is with the Elgato, or possibly the source.
Is the Elgato overheating? How is the case ventilation?
Have you cleaned the PCIe slot?
Try using a DeckLink Quad HDMI in your PCIe 4.0 x8 slot to see if the problem persists.
The stuttering only happens in full-screen mode (via GPU); have you tested outputting to an external display?

This could be a jitter issue due to memory thrashing or DPC issues. Running a tool like latency monitor from sys internals can help shed some light.

Here are some settings that can help (I know you already did some of them but just adding for others) and if not they most likely should be set especially on a system like yours and what you’re pushing through it. Also make sure your clock is set with NTP or synched up prior. Clock drift can add to some of these timing issues.

quick check list:
Basic Power Settings: Ultimate Performance or equivalent
Make sure it’s on your highest plan. Like ultimate or performance for the system. You don’t want the CPU downclocking at all. When it’s at a low 10% or so for a while it might downclock and now you would have DPC and jitter issues while its transitioning

Also under power options disable selective suspend on USB

Priority Control of Threads: Background Services
You need to reboot after this
Regex: reg add "HKLM\SYSTEM\CurrentControlSet\Control\PriorityControl" /v "Win32PrioritySeparation" /t REG_DWORD /d 24 /f
or
Win +r and type sysdm.cpl then advanced settings under the performance box switch to advanced and change to background Services and Apply.

Long story short this doesn’t give background services priority over programs and aapplications, it just doesn’t downgrade them if they are not front and focused. Has to do with P cores and E cores. vMix has a lot of threads doing import things including audio and NDI and down shifting them when it wants to would cause issues regardless of your CPU %. It’s a huge improvement with real-time apps and live streaming where timing is everything.

Then turn on GameMode

Disable VBS!
Core Isolation / Memory Integrity (VBS)
Windows Security > Device Security > Core isolation details

For windows 11 turn on Hardware-accelerated GPU scheduling and for Windows 10 with RTX 30xx+ cards turn it OFF

Make sure vMix is using your RTX video card by assigning itt o it under Graphics Performance Preferences
Make sure NVIDIA is running at Maximum Performance under its power usage settings

In vMix make sure input performance mode is turned OFF if you have rebar enabled. Makes ure all your audio and frame rates match across the board and all those optimization tips that vmix has in their FAQ

There are a ton more but these usually get the job done. Depending on if your windows 11 or windows 10 matters what is set and I have about 20 more optimization tips but the real score is doing it from the beginning and that’s froma fresh install through a distribution package that can turn a bunch of the telemetry and services that can cause DPC and other thrashing that no one really notices unless you are running realk time video and or audio packages. Anyway it’s been a while but let me know what you found or if this helped or if you need more direction on some of them.