Because we have hung with that friend. We have had lengthy discussions with that friend about the things we would do if we had a turkey baster, welder's mask, lacrosse stick and ten minutes alone with J. Lo's ass. We know more than we can bear exactly how much of a man our friend is, and all of the filthy things that entails.
And while I haven't had a lot of sister experience, I do have a younger brother, and I can confidently say I think of him as a tender flower, gently blossoming into womanhood. And if my friends tried to get with that, shit would be ON.
It's definitely an emotional reaction rather than a logical one. After all, you'd imagine that we at least think of our friends as good guys, and would therefore rather have THEM date our sister than some total stranger. But for reasons that are only clear to men, behavioral psychologists, and chimps beating the crap out of other chimps, this is not the case.
Another aspect of it is the concern that we'll somehow lose our friend to you. Guys tend to go a little crazy when they start a new relationship, and let's be honest: whatever enjoyment we can provide said friend, we will NEVER match the level of enjoyment you can dish out. Or if we did, the "friend" label would kind of become obsolete.
I'm not one to stand in the way of true love, but anyone interested in dating a friend of a sibling should be prepared for things to get awkward, angsty, weepy, or all of the above.