Since I often work with problem horses, many of them dangerous, I haven't actually met a horse yet that I've considered a hopeless cause. I've come close a time or two, but thankfully I've always had the answer come to me at just that pivotal moment.
I think what's more important is knowing 'your' limitations. Knowing when you need help and doing all in your power to get that help before things get out of hand.
So far every horse problem I've encountered has turned out to have originated as a people problem, or a physical ailment, and therefore once you take away the people problem, or fix the ailment, the horse eventually turns around. Admittedly, this can take weeks, months or even years so you have to be committed and in it for the long haul.
There can be the rare case where the horse has some sort of neurological or chemical imbalance, but even for those individuals there are many things that can be tried.
If this is a simple case of you don't click with a horse, then there's really no sense trying to make the relationship work. Sell or give the horse to someone who suits that horse better. And don't feel defeated or down on yourself if you have to do it. View it as a learning experience and avoid making that mistake again. Be sure though to disclose all behavior issues to the new party.
I happen to love the challenge of a difficult horse. Those individuals have been my greatest teachers. There's not one experience, even the painful ones, that I would change. I do, however, understand that that's not for everyone, so don't keep living a frustrating situation because it helps neither of you.