What you describe is not
normal in my experience, at least for trail riding. When I have a horse that does that on the trail, something is bothering him/her. It might be a sore back, or another horse nearby that he doesn't like, or maybe (if I'm on a mare) she's coming in season and getting "mare-ish", or possibly the bridle isn't fitting properly. On the trail, what I want to see is the horse's ears constantly scanning around, listening, and when I say something I want to see at least one ear coming back to listen to me, then back to scanning around. That indicates the horse is calm and relaxed.
Riding in the arena during training and general workouts, is a bit different. Then the horse often has it's ears tuned
to me, rather than the trail (ears on me, eyes on the road ahead). The horse's ears should be turned toward whatever it's paying attention to at the moment. Often it will have one ear tuned to me and the other listening or scanning for other things. If its ears are simply turned back, though, likely something is bothering it.
Make sure your saddle pad and cinch are clean, and that his back is smooth and healthy when saddling, and make sure your saddle isn't fitting badly. It could be you have a short-backed horse and the saddle bars are too long, or that you have a narrow-withered horse and the saddle gullet is hitting the withers, or something like that.
Then again, just like people, some horses are just plain ornery, and that could be the case with yours.