I know how you feel.
I was the same way with my ex-trainer's stallion. He did dressage stuff I wasn't able to do on other horses (i.e. get on him with just a halter, and we'd do just about everything - from piaffes to extended trots to tempis to jumping.. I even took some kids for their first canter on him (with a bridle of course)) he is simply amazing. We clicked. He wouldn't do any of this stuff for anyone else, so it was just... ahh I can't even describe it.
Anyways, I was going to buy him.. but he wouldn't pass a vet check because of a stifle injury he sustained when he was 3 (back feet slid out from under him in the winter) so he wasn't sound trotting out after flexion tests. However, never took a step lame while he was exercised. And exercise actually made the injury better. He was still able to jump over 4' and excecute all the movements I wanted him to in dressage... but my parents would never buy a horse that wouldn't pass a vet check. hah, this was back when my parents supported my horse habit.
He's now gelded and sitting in a paddock doing nothing.
It's breaking my heart, even now. I go visit him every once in a while when I can, but sometimes I can't even stand doing that, but I know he needs some love.
I would buy him today if I had the finances to... but I can't take that chance right now because I wouldn't be able to sell him fast if I absolutely needed to if I came into a financial crisis. It KILLS me. He's only 14 or so now...
So I know what you're talking about, and I still get upset over thinking about it.