Tearing your family apart is only of secondary concern. The primary concern is its ability to tear you apart. Perhaps you haven't felt its specific impact in your day to day life yet, but at some point, this deep trauma - this enormous violation - will undoubtedly begin wreaking emotional havoc in your life. It will tear you apart from the you you wish to be.

That's the bad news. The good news is you have a strong supporter in your boyfriend and from the tone of your question a long adult life ahead of you. Some guys would be bugged out by this and somehow make you feel even more isolated and damaged. On the contrary, your dude is leading the charge in helping you take healthy next steps.

You should trust your instincts, but also be aware that a portion of our instincts - what feels right - is more habit than anything else. For example, part of the psychology you may be unwittingly playing into is an instinct to protect your brother or parents at the direct cost of your own mental health. Maybe it feels better to absorb and make light of the incest, but I wonder, then, why it's now surfaced in your relationship. For one reason or another it's out of the shadows. I strongly suggest continuing to shed light on it.

This doesn't mean calling a family meeting tonight at 8PM. But it does mean, admitting that portion of your history deserves your time and some guidance/context/framework to process it all out (therapy, counseling, etc). If your family is worth protecting (as I assume they are), then they're also naturally in the business of truth and loving you.

-----------> Here might be a good place to start you off:
1.800.65. HOPE