How old is this gelding? How long did he race, and if he was retired early, do you know why? How many other people ride him? The life of a racing horse is not a pleasant one, and they bred to be a bit more excitable naturally. Your energy will influence this horse greatly and I completely understand that you are uneasy. You've gotten hurt before, and horse related accidents happen often, so be very careful. When you tie him up for grooming start by massaging him. Find out what he likes. Some horses love their faces to be caressed, some their belly to be scratched, some like their muscles massaged. See what gets him to respond and get in a relaxed state, then start the grooming and tacking up after his head is low and he is relaxed. Talk to him at the same time in a low calm voice, and let your voice be a calming white noise. Is there a particular part of the grooming or tacking that he seems uneasy?
Do you lunge him prior to riding? It can help to get out the initial jitters. Wait until he is licking his lips and lowering his head until you end the session. Sort of like when a person goes for a brisk walk or jog, when you are done you feel so much more relaxed and at peace.
Exposure training will help with spooking on the trails or outside of the arena. Can you put foreign objects in with him (yoga exercise ball, pool noodles hanging from rope, plastic bags tied to the fence, chairs, large plastic kid play equipment, etc.)? Tie a plastic bag to the end of a whip and hold him loosely on a lung line and start moving the bag. Slowly progress over several sessions to get to the point where you can rub him with the bag and he is not phased.
Some horses are just spooky naturally so above all be safe, wear a helmet, ride with a saddle, use quick release stirrups, and don't let your guard down for a moment. I recommend not riding alone or at least have a cell phone on you for emergencies. Use common sense and if you feel this horse is not a good fit for you then find one that makes you feel safe. Good luck.