Spend more time asking for gives as you ride. The horse needs to know who is the boss, and it is better to ask for lots of small gives vs having to be the big mean boss when he blows up.
If you keep him busy with lots of small request. He's not going to have the time to blow up. Now don't be a nag. You hate when your mother nagged you to clean the room. Buts lots of small request and atta boys.
Circle a sagebrush, speed up, half halt, full stop, back up, move out, circle left, circle right. Drop his nose, give and touch his nose to your knee on the right then give to the left. Nothing really drastic or that leaves him way behind the group. But commands that you know he knows how to obey. Just get him used to saying yes to everything you ask.
If he is busy, he won't have time to worry about a shade boulder or being scared about a shadow behind a tree.
I've rarely found it to a saddle problem, Unless there is severe pinching or burr under the saddle. Most problems are the horse just being resentfull of not having his way. Just give him a job and keep him focused and he will behave better.