I’d be more annoyed at your friends, in this particular situation. You’re a grown woman. Yeah, you may have some awkward history with this guy, but by the same token, it’s not their job to protect you from your personal choices.
Talk to them about this: Feeling protective is normal, but I suspect none of them are your dad, and even if one is, they all still need to respect that you’re your own person. As for the guy in question… I’d let it lie unless he brings it up. There’s no need for him to think a squad of guys is out to kick his ass.