With everything you have said taken into consideration, Nuala, I have to reccommend that you stop riding him, or at least reduce it to a few laps walking around the pen once or twice a week between groundwork sessions.
I'm also going to say that you should learn as much as you can about starting horses, and make a plan and a set of goals that will compliment your horse and build his confidence before real training sets in when he's older.
On another forum, I had put together a list of goals for a 3 year old horse by the end of it's third year:
-A good stop and back up
-Following the rein
-Riding with progressively less rein and more leg.
-Giving to the rein and flexing both left and right
-moving out comfortably when asked, no repetitive kicking asking to go.
-controlled trot, preferably on loose rein
-Making circles and serpentines at the walk without fighting the bit
-moving the front end,
-moving the hind-quarters
All of the bolded ones can be taught and mastered on the ground first, along with many, many more things that I could have listed.
In addition, you can factor in trail rides if you pony him with another horse on trails.