A beginner friendly horse is just what you need. Look for one a little older...at least 8 or so. If its been trail ridden extensively, that's probably good...as long as it was good on the trails. Horses used in lesson programs are a good way to go was well. They are usually forgiving, easy to work with, and know the basics. Id look at some barns with reputations for having well mannered horses. Since youll be riding for fun, maybe a horse that you can let sit for a month, then pull out of the field and ride without any worry is your ideal match. We were lucky to find one of those last year, she's an older QH mare and just easy as pie. Hopefully, you can find a good one under 5k. If you look older, they are usually a little less because of age, but that's not always a bad thing ya know. Be wary of any free or really inexpensive horses, they may come with some baggage. So shop around, ask as many questions as possible (regardless of how silly they may sound), go back to see it 2 or 3 times at various times of day to see how it behaves (feeding time, turn in out, working, etc) if you can. Take pictures, videos, etc to help you choose. And don't forget to handle them, and ride them! :)
Besides budgeting, unless you're keeping them at home there isnt much else to do. Make sure the horse will be at a barn with quality care. Don't forget about the farrier and vet visits, and the rotational deworming (hopefully your barn will keep you UTD on all that to know when things are happening). Most importantly, make sure itll be a horse you're going to be happy with and enjoy. Buying the wrong horse can lead to endless problems. Prepare yourself for all the fun, and also the possibilty of illness, injury, or death. If you can handle that, you're good to go.
Best of luck!