Yo, your man sounds like a bit of a Silly Billy, shall we say. His working logic on sexuality and the gender specific do's and don'ts would make Don Draper blush. Yes, fantasies on both sides of the genital aisle are perfectly normal.
The real kicker for me, is that he's so insecure that the thought of you thinking a celebrity - someone you (sorry) have .0000000006% chance of ever boning - is a threat doesn't bode well for his emotional constitution. I for one, feel more comfortable knowing my lady wants to lick Ryan Gosling's tushie than having her mention that one of my friends is cute or something like that.
But, that's on me, not her. Both are harmless. Both are not unreasonable expressions. Neither are declarations of action or intent. Your man shouldn't hold you to ideals he can't embrace. Tell him your fantasy life is a one woman operation and its guest list is private. So go hog wild, get your fantasy freak on with Tom Brady, add Bill Belechick in for good measure, and, sh*t, why not the rest of the offensive line when you're feeling adventurous.
Funny Guy
My wife and I mutually agree this is fine. After all, American celebrities are out of my reach (but since she's upper class, she can meet Chinese ones). Not only that, neither one of us has problems complimenting other people, and in the case of celebrities, joking that a particular one is our girlfriend/boyfriend.
One particular case that would probably make most men crazy, we were video chatting while I was in an Air Force office, one of the airmen sat behind me, and when my wife saw him, she immediately asked me who's that handsome soldier behind me. Far from bothering me, I told her who he was and even turned to him and told him my wife finds him handsome.
So it all depends how how close you are.
Funny Guy, you nailed it. My guy thinks it's quite possible for me to be with a professional athlete. It's a delusion, but at least it's not mine. I've been banned from ever meeting Tom Brady, but Aaron Rodgers is OK. I guess my interest in him has been deemed more professional!!