It makes sense that he would say it to cover his own ass, but the statement is crap.
There is no excuse for abuse. Whether or not you frustrate him is irrelevant: I’m sure he frustrates you, but you don’t respond to him the way he responds to you, I assume. And you can’t control how he feels, anyway. If he gets frustrated, it’s because he let himself get frustrated. Our feelings are our own: we can’t pin them on other people, because other people don’t have to power to make us feel something. So, his argument that you cause him to react the way he does is ridiculous.
Couples can argue without being abusive. There are rules to fighting (read this). He chooses to respond to you abusively, which is never necessary or acceptable. Don’t stand for it. The next time he starts berating you, leave the room or the house. If it’s on the phone, just hang up. Tell him you are happy to discuss your differences calmly and civilly, but you won’t accept being treated with abuse (and cruelty is a form of abuse). Then stick by your guns until he learns how to fight fairly.
Good luck. Thanks for the question.