I am considering building another machine that can take 8 SDI inputs via DeckLink Quad cards, include some additional video files, graphics, etc... and make use of MPEG-2 TS recording at the same time. I have a Core i7 CPU which still has plenty of headroom on my existing system.

Currently, I have a system with an ASUS GTX650TI-OC-2GD5 GeForce GTX 650, render time in vMix in a 29.97p project jumps to around 14ms with 8 SDI inputs loaded into it. About 25ms with other graphics and video files loaded into it (at this point it isn't dropping frames but is right close to the edge). Once I begin recording, the render time jumps over the threshhold of dropping frames and the FPS comes down to around 25 with 50-80ms render times.

I'd like to get a graphics card in here that can handle all this with some extra headroom, and I'm willing to pay if I know it would. Is a GTX 770 series around what I'd need? GTX 780? How will core clock speed impact vMix render time? GPU memory size? Do CUDA cores play a role at all? I know most graphics cards at this level are designed for gaming, not vMix.

I've also concluded it is better to run vMix in Windows Aero mode because it fixes sync/tearing problems, and in testing it appears to offer inconclusive performance benefits/hits.

I think these are very good questions (just pondered about them myself when choosing a new laptop).

Martin (in the FAQ and in some announcements in Studiotech.tv), as well as other users, has some general advice. My conclusion is that to really know one has to try. And then share one's experiences in the forum ;)