I am not sure if anyone has said this yet, but board would be a pretty big expense.
Depending on your area, what type of board(full board, self care, pasture etc.) and where, it could cost you practically nothing or several hundred dollars a month. I've boarded my horses at places that were $500-$800 a month just for self care pasture board. Now we don't pay board because they are at our house.
Also, tack. Cheap saddles don't last, normally you pay for what you get. Most decent bridles are about $40-60 and saddles are upwards of $300. Plus saddle pads(plural because you can't have just one lol) are about $80+ if you ride western and want a decent one.
If you haven't been around horses much, I STRONGLY recommend taking lessons from a reputable trainer. Horses are "money eating pigs" as my dad says, so if you like them but haven't been around them much your perspective could change. One of my friends talked about horses non stop for about a month, all he did was say how he wanted "fresians and gypse vanners to show with" until he came to ride with me. I made him do all of the heavy lifting and dirty work, the stuff you HAVE to do when you have horses. He hasn't said a word about horses since haha :)
I hope you don't think I am trying to deter you, horses are amazing animals! But, with horses comes responsibility, time, money, and HARD work! They are extremely rewarding, but expensive and time consuming.