Other people have no business freaking out over your horse. Obviously he's fine, and their perceptions of his behavior PROBABLY make it worse. I speak from experience. If I'm confident, my horse is confident. If I'm confident and then someone says omg he spooked!! HE'S SO BIG OMG then he starts getting riled up and he starts dancing and spooking more, at barrels at leaves flying, at me sneezing or coughing.
But what you can do to help, is have these people help brush your horse or whatnot to help THEM get over it.
And then do some simple head lowering exercises, releasing your horse's jaw via a finger in the corner of their mouth. Massaging the muscles along their neck and back (not the spine though.) Do take him for little walks and sack him out to things to build HIS confidence up.
Don't let other people frighten your horse.. horses are so quick to pick up on emotions, that's probably a big part of your problem.
"Strength is the ability to use a muscle without tension"