Do you mean that when you try to walk back toward the saddle, such as when you want to make an adjustment, or to prepare to mount, the hrose backs away from you instead of standing still?
If so, I would move a bit more quickly and out and around, in a bit of an arc, toward his midsection. IF he tries to back, then go with him, with your hand up on the saddle. Move calmly, sothat you are only followin him, not adding any push to his movement. you can say "ho, ho" if you wish, stay calm and stay focussed in that you are asking him to stand off to your side, not back away from you. He will stp backing in a bit.
I would not punish but I would just stay with him ad let him know that he is just working harder for no good reason. Once he stops, you just say a simple "good boy" and go immediately right to the business you started out to do.
When you have clear intention in your mind and body that you approaching his side is to get near the saddle, VS to back him up, he will read your intention as clear as day.