23 is definitely not too late to start!
My mother began riding again at 45 after a 30 year break. She dragged me into it, and I took lessons and rode for years. I hated every minute of it. I didn't enjoy riding until i quit for 10 years and came back at 18. I've only been back at it for 3 years (2, since I have been at university without my horse this year), and have learned more about horses in the 2 years of being passionate and self-motivated about them then I did in years of lessons and being motivated by ribbons and nonsense. I am happy I started 'late'. The best part? I found my dream horse when I was 18 and he was 5. With how long horses live, he will be with me into my 30's, 40's, and maybe longer.
"Putting pads on a gaited horse is like putting a mask on a beautiful woman."