Short answer - yes, you should be worried.
Long distance relationships are always tricky. But perhaps the trickiest part about it is that it's very hard to determine if there is a problem forming until it is too late. Even if you talk on the phone every day, it's not the same as being in the same room and seeing the look in your partner's eyes when they speak. A lot can change between once-a-month visits, and perhaps something already has.
In other words, in long distance relationships, you often don't know if something bad has happened until it is too late.
Your boyfriend is coming to your town to go to a wedding and he somehow can't find any time to see you. That's a major problem. He's been able to find the time to visit in the past, but he somehow couldn't find an extra day when he's already there? Ask him why. Let him know that you want to see him. Remind him that, in a long distance relationship, both people need to make time for each other. My guess is that when you have this conversation with him, he's going to admit that his feelings for you have changed. Or that he doesn't want to be in a relationship with someone who lives far away.
Either way, I think you're on the wrong side of his paradigm shift, and you're about to get some bad news.