Lots of good advice here :)
I bought a horse when I was in college and broke, and I was successful at it. Horses are expensive, but they are incredibly worth it. The biggest thing to have in mind is that if money is tight, you'll have to sacrifice something. I was unable to buy new clothes or go on crazy spring break vacations like others, but I don't regret it at all; every second with more horse was a million times more precious.
I also got away with borrowing lots of things. I had my own grooming kit (I bought an inexpensive kit and slowly upgraded as I could afford it), and my own saddle and bridle. I got the saddle on eBay, and the bridle, bit, and accessories from tack catalogs--I always kept my eye out for sales and coupons. A big plus to being able to borrow is that you can try out things to see if they work for you and your horse. I don't own a thousand bits like people tend to do lol, because I knew what worked for my horse after trying about 4 different ones borrowed from people at my barn.
Always keep some cash on hand for emergencies, and a credit card kept at a zero balance can help too. Equine veterinary care outside of routine vaccinations and tests can shoot up into the thousands really quickly.
When you're cutting costs, never scrimp on something that relates to horse-health. It will most likely lead to big vet bills later! Generally, I made sure my horse had what he had to have, and then made a list of my *wants* and I bought things as I could afford them. It was a pretty good self-compromise.