I guess I don't think that four days is all that long, especially if this is one of your first dates with the guy. You can't assume he's not interested. Maybe he just doesn't want to appear overeager. People play so many games anymore and try to follow so many damned rules, it's a wonder that anyone ever gets together at all.
I don't know why he hasn't called, but if you like the guy and want to date him again, sure, why not contact him? Who knows, he might be sitting there thinking you don't like him either because he didn't hear from you after the date. I know it's traditionally the guy's move to make contact, but we have to get past that. I see nothing wrong with reaching out to him if you want to keep dating him. The worst that can happen is he'll say no thanks, which is the equivalent of not hearing from him again anyway. At least that way you'll know for sure.
If you're really worried about making a wrong move, you could give it another day or two, but I wouldn't wait too long.
Thanks for the question.
Just say thank you and you had a good time... You're not taking any risks saying that, and if he's interested you'll know by his answer (hint: if he just replies "sure", he's not interested).