I can't say if it's wrong because I get the distinct impression that his lack of ambition is just an excuse for you to leave. I have a hunch that you're just young and tired of being married, probably because you signed your question "Young And Tired Of Being Married." What you describe just doesn't sound enough like a deal-breaker to me.
So he had a bad time at his job and he's been depressed and moping for three years. That's a long time, I agree, and I'm sure it is annoying. I don't know exactly what happened with his job, but are you certain that he has given up on his dreams? Don't you think you should give him more than three years? Pursuing dreams can take a lifetime, you know. Has he really given up or is he just in a funk right now? Have you done anything to try to help him get back on track?
My sarcastic gut reaction when I read your question was this: yes, by all means, bail when the going gets tough. All that stuff about for better or for worse, in good times and bad, 'til death do us part, yadda yadda yadda, forget all that and just beat it because it's not fun anymore. But I can't be that dogmatic about it because I don't know how bad the situation is and how incompatible you two really are.
I suspect that if he jumped out of bed tomorrow with a new lease on life and started chasing his dreams, your feelings wouldn't change. I think you married someone you probably didn't know all that well, and/or you're just over this guy. If I'm right, asking you to give him more time to snap out of his funk isn't going to make any difference in the end. I think you've made up your mind and you're just looking for someone to tell you it's okay.
I can't tell you to stay in a situation where you're unhappy and certain you will remain so. If you're over it, you're over it, and you have a right to bail. Personally, I think it's premature, but I don't have all the facts about your situation. All I can suggest to you is that you examine your motives before you decide to walk out the door. If you must leave, at least know exactly why you are leaving. That's only fair to both of you.
Thanks for the question.