Expense wise, I would start with a quote from Clinton Anderson. "There are only three ways to get a trained horse: buy it trained, pay for training, or train it yourself."
Ultimately you will train your horse yourself whether you want to or not, and early on most of your "training" will be to undo whatever training your horse has already had. Eventually you must learn at least enough to keep that from happening.
But the point is that when you buy a horse, keep in mind that a thousand dollar horse with little training might end up costing more than a $3,000 horse that is trained, and it will sure be a lot less fun. If you buy a horse that is ill-trained you might want to budget for hospital visits.
A good saddle will cost you at least $1,000 - $1,500, even used. That will be your biggest tack expense, unless you are the type who tries to solve problems by buying a different bit every month.
Horse, board, trainer, farrier, vet, riding instructor, feed, tack would be the expenses you'll want to put a number on for your area. For example, I had the vet come out this spring and vaccinate both my horses for rabies, tetanus, EW and West Nile, and do dental exams. Complete with farm call the total was $154.00. I've averaged less than $200/year on vet bills for the last 17 years. So you need to count the cost for where you live.