Have you thought about leasing? It really is a great option and often more affordable than owning, but guarantees the horse a home when you leave.
If there is no opportunity available to lease, then I can see why you would want to own, but remember to plan in advance a long time to find the horse a home when it has to be sold and be prepared to take a loss.
As far as feeling guilty for buying other things, the only person that's harming is yourself. If you feel the need to buy those things, it's your money and your choice. Each time you are buying something, hopefully you are thinking about whether it's worth putting off getting another horse even further so that you don't wake up years later regretting it all. If you don't feel you would regret it, why would you feel guilty? I'm a bit confused about that.