I believe the issue is that vMix has no ability to render a native 1080x1920 raster, which I'm detailing here:
https://forums.vmix.com/...t--vertical--Resolutions

If your vMix canvas is 1920x1080, rotating and scaling inputs to fit will give you 607.5x1080 pixels (9:16), which will definitely look worse as you're losing almost half the vertical pixels.

If your vMix canvas is 3840x2160, you are forced to scale all the way up to 1215x2160 (9:16). Then it gets scaled back down to 1080x1920 using your encoding settings (I do 3412x1920, then crop to vertical. 3413 is closer to the correct integer but vMix complains). This also results in a loss of sharpness, though not as dramatic as the first example.

vMix has a 3840x1920 option, but I'm not sure what it's for as it's not a 16:9 ratio, and extensive testing has shown me that the input is actually treated as 2160p, so the scaling issues are still there, compounded by aspect ratio problems. In the process of testing this I've found that vMix's max custom Fullscreen resolution is somewhere around 3300 wide, so we also can't get correctly scaled vertical Fullscreen outputs.

OBS allows you to set a canvas size of 1080x1920, so you can maintain pixel-to-pixel accuracy through the entire chain for the highest possible quality. This would be the ideal fix for vMix to implement.

We do several vertical streams per month for large corporate clients and are likely going to have to switch to OBS for encoding in order to maintain native resolution.