Good job taking hi out!!
Unfortunately this is often the case with ottb's, at the races they're so hyped up, they know they're going to gallop flat out in a bunch of other horses so it's expected that they'll be quite a handful at their few few outings.
The best thing for him will be to take him to as many outings as you possible can. Trail rides, take him to a friends house for a ride with them, shows etc. Anything!! Just get him out as much as you can to give him exposure. Eventually he'll come to the realisation that he's not going to have to run, and will start to settle.
I prefer to just take mine to a few led shows before getting into the ridden ones. This avoids what you had problems with, horse gets tense, then you get tense and it's a vicious circle which results in an overall tense outing for the horse. If you enter some led classes, or even just take him along to the comp ground for the day that will be of enormous benefit to both of you. Get him off the float, walk him around the grounds, do as much in hand/ground work as you can to get his brain focussed and back between his ears, then go back to the float, tie him up with a hay bag and sit and watch the show. Give him a few hours to settle at the float, and if he seems content with being there, and your nerves have gone, saddle him up, take him out to a quiet corner of the grounds, ground work him again to get his focus and then hop on and go for a nice sedate walk letting him take in all the sights and sounds.
That's my routine for at least the first few outings, it does depend on the horse, others settle quicker than some, but I won't enter a ridden comp until the horse is settled going out.