So, because his wife sticks around, it is OK for you to screw her man? Not quite sure which of the pair of you is dumber, here. Her, probably, but you are running a very close second. You are darn tootin’ it’s wrong.
To answer your question, what makes you think he thinks anything of you at all? After all, he certainly doesn’t think at all about the woman he actually, you know, freaking married!
Stop it. Have some bloody pride.