I have a vague recollection that in the past, with multi-channel Blackmagic cards like this one, it was best to use the lowest numbered channels for output (Ch1, or Ch1+2) if the other channels are being used as inputs. Just something you might wish to try, in case it applies here (despite not being apparent from the error message you're seeing).

Hello, patrickfrog
I have also Decklink Quad (not 2nd generation), but I cannot run input from cameras. How you ajust this capture card - easy, without any specific settings? Because in Supported Hardware list Quad1 not included. But another card with 4 inputs I don't have now :(

quantum,

To make my inputs work, I first made sure everything was compatible resolution and framerate-wise. Here is the spec sheet for the card:
DeckLinkQuad.png (154kb) downloaded 3 time(s).

In my case, I had cameras with both an HDMI and SDI output but I was unable to use the SDI output because its specs did not match exactly with the card unfortunately. So, I ended up using a BlackMagic HDMI to SDI converter which works nicely.

Second, I would make sure you are getting any video by using the BlackMagic media express software that came installed with the driver. This will tell you if it is a VMix issue or some other problem.

Finally, when you add the input to VMix, make sure all the resolutions and framerates match your camera and on of the valid settings from the specs above.

That actually leads me into the solution for my problem. It turns out that VMix will throw this error if it is not running in a resolution and framerate compatible with the output card. I was running in 720p29.97 which is not compatible with the DeckLink Quad. I switched to 1080p29.97 and was able to get video out!!