I don't think you are crazy. Brave, yes. Crazy, no. And note to commenters - if I find a guy hitting any gal, my usual reaction is to put the sod in traction - OK?
I'll let you in on a wee secret. All guys have temper issues. It goes with the whole "being a human being" aspect of life. Every last one of them. And yes, it can be worrying. What will you accidentally do that triggers the ragingly violent beast that lurks in the souls of all men? Honestly, probably nothing you ever do will trigger it.
Most guys, apart from the completely Neanderthal jerks, reach for the physical violence hammer when they have absolutely nothing else left to respond with. It is both disgusting and completely wrong, but it does happen. Even I have been known to punch holes in doors or walls on occasion out of sheer frustration, though never with an audience. I have far too much self respect for that crap. Shy, low self esteem guys do tend not to be violent at all, though, as they simply know it doesn't work out well.
You might want to, very gently, find out actually why he hit his ex. Get the full story, before you start worrying too much and pulling the plug. For all you know at the moment, you may find yourself wishing he had hit her much harder.