There is absolutely nothing wrong with having a dream.
However for a dream as big as the olympics (and I think just about every kid that rides wants to go to the Olympics), you can't be dissapointed if you don't make it.
It is not about when a person started riding, but the dedication, talent, MONEY and time that they have available to them. Look at how many professional riders are around in both jumping and dressage. There are SO many extremely talented riders that still don't make the cut.
There is an enormous amount of money involved, not just in going to the Olympics itself, but in the preparation just to be somewhat competitive.
Unless you have a VERY talented horse (and in terms of Dressage, we're talking something imported from Europe, or bred with the best European bloodlines, which will cost a very pretty penny. The horses that compete at the Olympics are worth into the hundreds of thousands of dollars, some even towards the million figure mark.
On top of that, the coaching for yourself, you will need to base yourself with a professional coach (not just old joe blow from around the corner than can ride a little bit) - which is generally going to cost well over $100 per session.
It'd be nice to represent your country at the Olympics, but unless you can get very rich, and be unbelievably talented and have multitudes of spare time to put into keeping multiple horses in work, and attending CDI's across the country and internationally, then try aiming your sights at something a little more achievable.