You might do some groundwork with him, to gain his respect if anything.
The spookiness issue definitely needs to be addressed. It was a silly thing for him to freak out about, but imagine if you had been on his back, and he wanted to back into a ravine or steep ditch? I've had that happen- it isn't fun! Any horse will benefit from being sacked out every once in a while. We have a 19-year-old, sane-as-they-come type of horse at my barn. He spooked at something one day, and the owner was just livid. She thought he was doing it to spite her, which is silly. Horses aren't humans. You can't put human emotions like jealousy and spite on a horse. There's usually, "Get the heck off me!", "Hmm, this isn't so bad." or, "You're irritating, but I'll tolerate it." They're not "plotting their revenge", lol!
You're doing fine!