Audio buses. Pure and simple.

Assign the pre-recorded material and the other person to a bus, this will be your monitoring bus. (A Mix-minus, as in you're receiving everything mixed minus your voice)

You really shouldn't need to monitor your own voice, for the reason you discovered. Latency. It will always throw you off.

Conversely, you should send your guest on Skype every bit of audio minus their voice. Assign you, and the material to another bus, and send that as your outgoing audio.

You shouldn't ever send "master" to any guest, because the chance of them hearing themselves back get higher. (You never want that)