Depends on the sort of sex you have.

If it is with people you are fond of (or love) then not at all. If it is simply scratching an itch with the first randomly available guy, you will likely have problems (well, you already have problems, but different ones).

Tangent time

Sex should always have emotion involved. At it's gutwrenching best, it is the way to get as close as possible to whoever you are having sex with.
I am aware that there are a lot of people out there, both guys and gals, who can have emotionless sex - mutual masturbation with a convenient warm body, if you will - but I really can't understand it. How on Earth can you respond to your partners cues if there is no empathy there?

So I asked one of the local working girls. If anyone would know, it would be a gal that has sex for a living. After a bit of chat to and fro, while she made doubly sure I wasn't a cop, she told me she really doesn't get that either.
Sure, many of her clients are one offs, which she describes more in terms of any mildly unpleasant household chore than anything else, but she has regulars who actively seek her out and who she is quite fond of. The ones she talks to before - and after.
The sex is always better for them if there is that emotional connection. Better for her too, she rarely has to fake it for the regulars, though guys really aren't her thing in her personal life. As far as she knows, the only gals who emotionally disconnect are the ones who have other problems already.

So there you are. An answer from an expert who basically says "Don't worry about it as long as you are reasonably sane." Hope you realise this answer cost me $40.

Well, time is money, isn't it.