I would not worry about the ears not being on you.
When I take my mare out, she'll only RARELY put her ears "on me". Most of the time they're forward. It just means she's paying attention to the trail and is focusing on her "job".
If we're riding with other horses, she'll have her ears on me more often.
I've found that her ears move the most when we're out alone. They'll usually stay pricked in front if we're leading a group of horses and when we're in the midst of a group of horses, ears ears are super relaxed and not really focusing on one thing.
My theory (through riding many different horses lead + farther back in line, seeing this again and again) is that when the horse is alone on the trail, they're looking out for any "danger" that might appear anywhere around them. When they have horses behind them, they're focusing on possible dangers in front and letting the horses behind them watch for side to side danger. If they're within a group of horses, the duty of danger-watch generally falls to the horses in front and back with minimal "input" from horses on the sides of the herd, so the middle horses are going to be the most "focused" on their riders.
Anyway, I would not worry about the ears. It's just his nature, as a horse, to be on the look out for danger. I would actually be more concerned, if it were me, when he doesn't have his ears out in front (when I see that, in my experience, the horse is likely to be "napping" and and possibly about to have a spook when he/she "tunes back in").
I think you have a bigger problem with him getting pissy when you ask him to turn around. I would focus on that.
Of course, my mare is not a huge fan of turning on the trail either. She's a firm believer that getting us wherever we're going on the trail is her "job" and she takes her job VERY seriously. However, she will do anything I ask her to do even if she's fussy about it. That's a rule I have with her - she MUST do what I tell her do do and she has the option of expressing her opinion through pinned ears or a head toss but she MUST do what I tell her to do without bucking/rearing/anything more violent than an ear pin/head toss.