Maybe. But you might be reading too much into his reaction. No offense, but this sounds like more of a personal quirk. Sometimes we touch hands with other people. It happens. Sometimes you accidentally touches a stranger’s hand when you both reach for the same pole on the subway. You say excuse me, and move on with your life. That’s what hand sanitizer is for. The fact that he’s okay with accidentally touching doesn’t automatically mean he wants to go steady or anything. He just doesn’t think that brief physical contact is that big of a deal.
The thing to pay attention to is whether he goes out of his way to touch you. Does he touch your arm when he has something important to tell you? Did his hand linger on yours when you accidentally touched? If you notice brief touches here and there, he might be flirting. Or he just might be a bit on the hands-y side.
You’re going to have step up your game here. Obsessing over whether or not a fleeting touch means anything will drive you crazy. Ask him to hang out as friends, and see what happens. Let things progress naturally. If he touches you again, accidentally or otherwise, don’t flinch. You just might end up holding hands this time.