I would definitely start by figuring out which type of riding you would like to do. You can ride either english or western. English disciplines include dressage, cross country, jumping, etc. Western disciplines include pole bending, barrel racing, etc. Most barns start beginners out riding western so they can work up to riding english but not all of them do.
Once you figure out that, you need to find a nice barn where you can take lessons. You want to start out slow. You can't just hop on a horse and go.
In your first lesson, you'll most likely learn the basics such as grooming, tacking up a horse, walking, and if they think you are ready, trotting.
You will probably start in either a round pen or on a lunge line.
Just remember to stay calm and ride in the right clothes. Open closed shoes are always the best thing you can wear. Also, jeans or breeches (if you have them..) would be good.
Once you get used to lessons and your trainer feels that you are ready, you may start half-leasing, leasing, etc. It is like owning but not as expensive and for people who want to learn everything about owning before jumping right in to owning a horse all on their own.
Just take your time with it and have fun!
Riding is amazing and I bet you will love it!
Best of luck :)