You sound exactly like me! I barrel raced and trail rode from 9-13, then moved. I started riding again when I was 18 on a horse trader ranch (I quit, the owner viewed the horses as money and not as living things), then I found random people to ride for off of craigslist.
Now, in 4 months when I get home from deployment, I will be getting my first horse at the age of 22. I'm not going to take lessons tho, I am self taught and pretty confident in my riding.
No, it is NEVER to late to get back into horses. The worst mistake someone could make in my opinion is NOT riding again! Because there is nothing like the bond between a human and their horses.
I support you 100%! There's not judgment in the horse world because of your age! Shoot, there are 8 year olds who ride better then me, but I swear they were born in the saddle lol! You'll do great!