How long have you had him? I ask this because it can take many months to rebuild trust in an "abused" horse.
If it has been recently, then the only answer is..SLOW, PATIENT, CONSISTENT.
What I specifically mean is this...Don't ask too much too soon.If your horse is a bit more at ease while walking, stay at that for a time, then when you know that has been consistently positive, ask to trot. Trot for a while, and so on...
You'll have to go very slowly, as he obviously has not had good experiences with people.Horse have in my opinion, the longest memory of any animal I know.
It will benefit you to handle from the ground more often, groom quietly, handle gently but firmly. I have been through this a few times, and what I have found is you need to slowly rebuild the trust in your horse, and we can't expect them to act in the time we think in...It can take a long time but, chances are, if you go slowly, only go for short sessions at a time (say half-hour a day for while under saddle, if that is where the problems mostly are) he will also slowly get used to your consistency, and with time, work out his trust issues.
As with any horse, your new key word is patience! Nothing good ever comes quick and easy! Good Luck to you -Lw