Hi tarzan, I am in the process of retraining an ex racer myself. Is he an OTTB or another breed? I started with lots and lots of groundwork. The round pen is your friend, if you have access to one. You really need to bond with him and get him to trust you and to establish that you are alpha. Dean and I did lots of hanging out, then work in the leadline, then learned to lunge and voice commands. I do hop on him for a little turn around the round pen but not for full rides yet.
Also you want to make sure he is pain free, think about having a chiropractor out. Make sure he doesnt have any ulcers.
Most of all keep in mind that he is only 3, and still a kid. Work on trust and do tons of ground work before you try to ride him, he needs to know you will take care of him and you need to know he will take care of you.
If he is an OTTB, look for the book Beyond the Track on amazon
. Good insight in there. :)