Hmm. Sounds to me like your +4 advantage has thrown him for a loop. Most guys aren't used to that; we're the ones who cat around, and women are supposed to be good little girls and only sleep with a few guys, or, at least, fewer than us. It's a classic double standard, leftover from the olden tymes when every man wanted to marry a virgin.
You told him your number; what else does he want? All the naughty little details of who, when, where, and what you did? No. Just no. He thinks he wants to know, but he doesn't, trust me. I've only asked one girlfriend about her sexual past, and as soon as she told me, I wished I hadn't. Now I'm married, and even after many years together, my wife and still I don't know all the details of our SBYs (Sex Before You), nor do we want to. We know enough, and the rest just doesn't matter. It's the past.
Ask him why he needs to know so badly. That's the real question here. Is he really that insecure about your relationship? Is he trying to determine whether or not he thinks you're a slut? Is it a control thing for him? What if he doesn't like what you tell him; is he going to break up with you? And how much detail would satisfy him anyway? Any tidbit you give him will spawn a hundred more questions.
Bottom line, it's the past, it's none of his business and the truth will only hurt him more. Tell Mister Man that you are focused on your future together, not the past, and refuse to say anymore about it. As my friends Kara and M.A. said, "Nothing good can come of it." Just say no. And ask him to let it go so you guys can get on with the relationship.