You push them away with your body language.
But the overall point of joining up is to let the horse initiate the contact with you because if they do that, and I'm hesitant to use this term because it's not necessarily true, they are "bonded" to you because it was their decision to make contact with you. And if a horse decides something on it's own, or you can get the horse to think it was their idea, it's more deeply ingrained into it's mind.
"And somewhere in the northwoods darkness a creature walks upright. And the best advice you may ever get is: Don't go out at night..."