I agree that they shouldn't be started being seriously ridden until they're at least 3, if not older. But I don't see anything wrong with hopping on them occasionally (meaning maybe once a month, no more often than that) starting at 2 1/2-3 years old. And by that I mean literally just hopping on their back, sitting on it for a few minutes (no more than 10), and just walking around or letting the horse wander.
Of course, you would need to have a horse calm enough to do that, which most people probably don't. But I'm pretty lucky, probably 90% of the horses I'm around are like that - you can just go out to their pasture, line them up with a fence/water trough, and hop on them with no saddle, bridle, halter, or anything.