You say this horse came from a "mixed bag". I wonder if he's just convinced the other shoe is going to drop whenever he's being ridden and reacting because he figures he might as well get it out of the way. Even though you're riding him in a very mild bit, maybe he was jerked in the mouth a lot and spurred frequently in his former job.
While you can't make a horse forget, you can be extremely consistent in his training to the point where he will eventually learn that your are not going to do anything to hurt him. You appear to have a quiet seat and hands which will work in your favor. I would say, be as quiet and consistent as you can be with him. I'm not saying never put a leg on him or ask for anything, but make sure every cue is always the same, X means X, not Y or Z if you know what I mean.
It sounds like he does start to tense up before he gets ready to blow. When that starts to happen, try frequent changes of direction. I'm talking never going straight until he's calm. Something about changes of direction seems to regain a horse's focus on it's rider. If you do trot or canter him in fields, again keep him on a large circle and change the gait frequently. Pay attention to his ears. If they are both forward, he doesn't know you're up there and might feel he's on his own for this safety. Once he's got an ear back on you, you're in business. Use it to ask for little things. Keep him guessing. He'll have to keep his attention on you rather than on acting up.
If he's a TB, check his tongue. He's probably got a big fat sausage tongue. If there are any marks on it, there's your answer on whether he ever had a rider with bad hands. Maybe try a bit with a low port for more tongue relief and switch to an egg butt cheek piece. Loose rings are great, but they don't provide much in the line of brakes in situations like yours since they'll just slide through his mouth when trying to do one rein stops.
If he does try to bolt, one rein stop. Then go back to walking serpentines, changes of direction until he's calm again. When he is calm, lots of pats and praise. When he's being aloof, ignore him. Ride him like all is right with the world and manage the situation. Good luck.
You just have to see your distance...you don't have to like it.