It sounds like people are just saying it might not be best to actually BUY a horse that you don't intend to keep for an extended period of time, which you have clearly said. A couple of years is not a very long time, and then you'd be stuck needing to find a home for your new horse all over again. Not trying to be harsh at all, just honest.
Have you thought about leasing? It really is a great option and often more affordable than owning, but guarantees the horse a home when you leave.
If there is no opportunity available to lease, then I can see why you would want to own, but remember to plan in advance a long time to find the horse a home when it has to be sold and be prepared to take a loss.
As far as feeling guilty for buying other things, the only person that's harming is yourself. If you feel the need to buy those things, it's your money and your choice. Each time you are buying something, hopefully you are thinking about whether it's worth putting off getting another horse even further so that you don't wake up years later regretting it all. If you don't feel you would regret it, why would you feel guilty? I'm a bit confused about that.