Open a command prompt as administrator and try this command: sfc /scannow

This will scan your protected systemfiles for missing or corrupted files and will try to fix any abnormalities.

code 0x5 is Access Denied, So it cannot load the file, or has no access to it, or is missing it.
RegSvr is used to add or update registry keys, so it might also mean it cannot write the registry update, because of the user not having rights to do so.
So also try to run as admin by right clicking and choosing the run as administrator option.

Also check if your virus scanner isn't keeping vmix from updating the registry.
You can also try to find out which key it wants to update within the registry and find that within regedit and look at the permissions on that key.
For this you should read the windows logs within the event manager.