I think that he may benefit from a riser pad. If you were to ride him western, he might have some bridging due to his high withers. This is easily filled in with a pad that is thicker in the middle and thinner on the edges. My husband's horse is getting along a few years in age, and she does much better with a "swayback" pad.
Your horse if fine. High withers can be good. They hold the saddle on.