He's a 4 year old TB - and a colt. Of course he's going to be a little leery, he's a young horse!
More miles, wet saddle blankents, and a good balanced rider who will not grab, pull or kick, and simply ride on.
I've got one who spooks and bolt out of genuine fear rather than naughtiness. He shakes like a leaf, sweat until he's white, and his heart thumps between my legs at a million miles an hour. With him, if he's scared and wanting to look at something, I need to grow some *jewells*, give my reins, put my leg on, and ride him forward. He tends to get over it if I don't make a fuss of it. I have also taught him a one rein stop, the times where his mind just goes, 9 times out of 10 I can bring him to a stop, pin his nose on my leg until he stops panicking, then we can go on our merry way like nothing happened.
The tension after a spook is very often a rider issue. Yes sometimes its the horse, sometimes its a combination of both partners feeding off each others tension. As a ride its our job to lead, and sometimes, when they spook, we become unnerved even if we don't think that we are. When this happens, we start to grip, exaggerate our movements, or not move at all. The horse is going to pick up on these changes straight away - and assume that it is right to be scared, if the leader is scared too.
Try taking a few deep breaths when the horse has spooked, close your eyes for a moment and go floppy. Then ride on. Forget that the spook ever happened.
Good luck with your colt, hopefully he loses the spook. Mines an 11 year old dumb-blood that I've only had for 8 months, and he's still got a massive spook in him.