I would suggest walking him up to it. Like, don't ride half way and walk the rest (unless it's quite a distance away)
He might feel more comfortable with someone on the ground, rather than on his back.
Also, don't think about that spot. You might be tensing up or expecting something to happen, if you think happy thoughts and visualize you and your horse just walking right by it might help :)