I totally know what you mean- I'm in just the same boat as you. I mostly get slack from horse people around here, though, who say I can just make it work with my horse. I've always known it wasn't right with us and I'm just trying to do what's best for the both of us! He's a great guy, but just not right for me. Horses are living, thinking animals with widely varied personalities, so not every horse is going to work well with every human!
It gets frustrating, but keep in mind that you're doing the right thing! No one knows your own situation better than you do.