You might look at Jitsi and/or Jitsi Video Bridge (http://jitsi.org) which are both open source projects.

Jitsi is a SIP/XMPP client that supports text, voice & video. It can be used to pass point-to-point video chat via common SIP services, like http://getonsip.com. I've used SIP phones for years, so SIP based video is a natural progression for me. There are various software & hardware end-points to choose from. Often I'll setup a connection to a soft client on a laptop, then capture the laptop's output into vMix using an HDMI capture card.

Jitsi Video Bridge is something of an alternative to Hangouts. It's a server side process that acts as a video router, switching between video streams according to who is actually talking. It's pure webrtc so there is no end-point software except Chrome.

In truth, there are many free WebRTC-based video chat services out there. Talky.io is one of the better ones. Blab.im is another. In trying to make everything easy& simple most of the WebRTC-based stuff won't give you any control of the media quality. That's where using your own instance of JVB can be advantageous.

Michael