Possibly a CPU issue while recording.

Do some testing if your CPU is over 60% (from taskmanager not Vmix) while recording and viewing with those settings I'd investigate that.

It would be the CPU being high "while recording" so it is basically dropping frames. It could also be an IO issue where your disk is a bottleneck and can't write data fast enough. Less likely, but more likely as the quality of your recording is increased.

Therefore look at the specs for your disk device. If it can only write @ 100mb/s (typical well configured HDD goes around that speed, some slower some faster of course) and you are trying to write @ 300mb/s because you are using multi-corder or very high bitrates you will have a problem.

IO problems can be solved with an SSD, which usually goes between 400-500mb/s or a RAID using many disks.

Basically it would be helpful if you listed the formats and settings you used for the recording and details about what your CPU was doing at the time. I suspect a performance issue, but if could well be something else.

Another place to look after that would be Video, if you are using some kind of H.264 acceleration on your video try switching it off. Video processing can be a massive help offloading some of the jobs that the CPU does. However if your video card is basic and not up to the task it can be slower than your CPU at encoding video.

I can't really get more detailed without more details.