As I've said before, guys, like other humans are complex mammals. There isn't one definitive reason why a man would think it hot to see his lady plowed by...not him. But, I will do my best to lay out some reasons this may make a man randy.

Option One
Starting with your own theory -- voyeurism. Being on the sidelines and having the unique vantage point of seeing your wife mid-sex can be a thrill.  It satisfies that itch of "If I could just be a fly on the wall." Your man sitting across the room is that fly. A six-foot fly with a boner, but none the less a curious bystander. 

Option Two
While I for one would rather see my wife having sex with some far uglier and smaller in the crotch troll than me, other men might enjoy seeing their woman engulfed by what they see as a "superman" or an ideal masculine force. Seeing a strange man having his way with his wife - in a way perhaps he cannot - satisfies some sort of carnal need. In other words, the other man serves as a proxy cock.  Twisting, lifting, spinning the woman and flopping his wang around in ways the boyfriend only wishes he could.

Option Three 
Though maybe more challenging for men to hear, there can be a level of homo-eroticism at play. Appreciating how this stranger looks or what he can do is a turn on in and of itself. This might not mean, "Oh I wish I was my lady in this scenario" but there are shades of envy that bleed into curiosity, arousal, and desire.
 
All of these theories to some extent play with the notion of control, intrigue, power, "humiliation" and trust. Like rough sex, playing dress-up or teasing and bondage, humans derive pleasure from manipulating situations, distorting their common realities and shaking things up. There is no right and wrong with this stuff -- provided all parties are consenting.  And, providing the original party is still able to party after the third party departs.