My colt had abuse issues, but I still had to be firm and assertive with him. Horses are looking for leadership. I would bet that he doesn't need you to be his buddy, but rather his leader.
So - be very precise and to the point. Be quick to release pressure when he gives in (it will be baby steps to begin with), but maintain pressure until he does give. Work with a carrot stick from the get go. Teach him that it is simply an extention of you and not to be feared. Don't beat him with it, yet don't be afraid to push his neck, ribs or rump over with it. Pet him with it after you release pressure.
Big key is to release ALL pressure as soon as he gives. So if you disengage the hindquarters and say you need to push with the carrot stick - as soon as he steps away 1. quite physically pushing, 2. take a single step back, 3. relax all of your body and facial features and let out your breath. You want him to learn to move and respond to the "pressure" of your presence, energy and body language.
Next big key is to have a plan and stick to it. Know where you are going and where you want him. So if you want to walk a straight line from point A to point B, look where you are going, put your hand and shoulders forward to cue that you are moving forward, tap with your carrot stick if he lags (as soon as he goes relax your energy and body and verbally say Good Boy!) When you get to point B pull your hands and shoulders back and stop. If he gets too close, bump with your elbow or ackkk at him, or stop, throw up yours hands till he steps off of you. Relax and continue on as normal
as soon as he is where is suppose to be.
Don't be wishy washy in what you want. Put yourself as the leader and he will follow you. If you are not direct, he will look at you as a useless human who cannot protect him and lead him.
In this type situation I will lean towards Parelli techniques. Clinton Anderson is my favorite though. Me personally, I watch and read, then go play with the techniques until I have put together what works for me and my horses.
In my colt's case - I spent an hour in the roundpen with him and worked on disengaging hindquarters, leading and desensitizing. The owner pretty much gave him to me, because I got further in one hour with him than she had in 6 months.
You can be his friend, but remember to be a leader and protector first.
Hope this helps