Just keep doing what you are doing; you have to desensitize him to your equipment, not overwhelm him with it.
If he wants to move away from you at any point, just drive him away, so you are still keeping his mind focused on you, but yet he is still working at a comfortable distance from you; you don't have to push him hard either, and I actually advise against really hard work, as you could ruin his joints, especially if it's a smaller round pen. When his ears, and nose, and eyes start coming in toward you again, then ask him to stop using your body; step in front of his motion. (You should drive from his rear, when asking him to move). When he stops, let him come up to you, or if he will allow it, go up to him. Then start rubbing him all over again...start introducing a soft brush too, if you can. Rub him with your lead rope, his halter, etc...work from middle to front; I wouldn't necessarly start from his rear, as he may get scared and kick at this early stage in the game, so keep yourself aligned with his belly, or shoulder at all times. Certainly work on his hind end, but make sure you are in a safe place so you don't get kicked.
When he is comfortable with the halter and lead rope being rubbed all over, then start slipping the halter on and off the nose; don't put it on yet, just let him get used to you sliding it on and off. You can certainly slip the leadrope around his neck at this stage too, to keep him from backing too far away...and if you have too, you can drive him in a circle around him on the lead, again, you don't have to drive him fast, just keep his feet moving so he can recollect his thoughts and concentration, and get over his fear. Don't rush haltering him; you want him to be relaxed in the process, and not wanting to swing his head, or try to evade it, because you put it on as he was pulling away...just take it slow...you've got lots of time; he's at a really nice age, even if he wasn't worked with before this...they are easily moulded.
"The ideal horseman has the courage of a lion, the patience of a saint, and the hands of a woman..."