I would say keep going. Unless you are on the heavier side I wouldn't worry to much, just keep things on the down low, like you have been, as he still has some growing to do:)
I normally start mine at 3, but several people start at 2.
Just my opinion.
I don't see a huge problem with keeping him in work once a week. Ruger was started at two, and even though I gave him a break while I worked with other horses I didn't feel it benefited him. He was so easy when he was first started, but after his break he got to be a real prick. I mean, it's up to the individual horse too. Some colts do just fine with time off. But, buttons get rusty...Often it takes me a couple weeks of being ridden to remind a colt of everys single cue he's ever been given with refinement.
There's no doubt leaving him to grow and mature will only be a benefit to him physically. I'm glad that you're even considering that as alot of people don't.
With regard to how much training he will lose, that depends somewhat on his personality. He's not likely to forget his training (I don't think most horses do) but he may be a little 'rusty' on some things at the start. If you're just riding him once a week now and he's doing fine, I speculate he's probably going to be alright once you get going again. If you decide to leave off riding, when you go to start working with him again treat it as if you're starting right over from scratch. It won't take as long as the first time and it gives him a refresher and helps the both of you get back into the swing of things.
Some outfits in my area who turn out really good horses start them in the fall of their two year old year, then turn them out until late spring. Then they do six weeks of riding and six weeks out until the fall. In the fall, the horses get some miles and turned out until the spring. At that time they are four and get either sold or put in the ranch string where they develop the remaining skills they need as life goes on.
Thanks for all the posts! They get me thinking for sure! For starters, Im weigh about 175 or so. Not terribly heavy, but no means light. To address his personality, he gets awful if I don't go out and do anything, I mean bad. Wont come up to you, will try to rush you for food, just impatient....but he gets an attitude if you don't at least do something with him, brush him, get him out, lunge him, ride him, ect, ect. So I think I may just ride him once a week to once every 2 weeks...and then give him some time off in between. Not very consistent though, but considering that's what I have been doing since I started riding, Im sure it will be fine! Thanks again!