You say he is stiff on the right side, I would consider getting him vet checked, then finding a physio, he may need some work on his muscle, god knows what they did to him on the track, he may be out on that side.
If he appears sore in any particular spot you could rub white viniger on it, don't rub on cuts) this will help ease soreness and tension.
I would also get his teeth checked, and has he been wormed.
The other thing prone to ex race horses is rolled fetlocks, this is where the fetlock roll out or in, in all gaits. I would ask a vet, or a farrier.
As far as building up muscles this takes time, work and good feed, but if I where you I would suggest you considering doing as I said.
You don't want this horse coming down on you while you r riding
Hope this helps