Sure, guys care about nipple size. Humans can be some scrutinizing mother-F'ers -- some more than others. There are guys who'll will break up with a supermodel because they find her nipple too large, or too low, or too long, or too dark, or too pink, or too... nippley. On the other hand, some guys can date a woman for five years and would be unable to identify her nipples in a police line-up.
But you are right, ratio and symmetry is often key. Small nips on small boobs may appear more attractive then penny sized nippers on 42Fs. At the very least I think that's what Janet Jackson and the porno industry has taught us; there is some sort of golden ratio of breast to nipple size.
But worry not; it sounds like your well in the norm -- far from a witches tit. Besides, most guys are psyched just to be touching a lady nipple. And by the time you're dating someone, they have more important things on their mind, like your ankle to calf ratio.