If you are comfortable uploading a picture of him, I would love to see him. It sounds like you have a very strong bond with him and that he does trust you. Not having a round pen or similar area to work with him, you can fix that on the cheap (with the money you saved by not giving it to that trainer LOL) with t-posts and cattle panels, or temporary electric posts (not charged) and thick nylon electric fencewire which is easy for him to see.
If you're able to fix up an area like that for the two of you, my gut feeling is to begin with using no tack at all, just a halter with reins, and simply get on him bareback and just do nothing but stand there. Do that for as long as you think is necessary before then asking him for the next baby step you want to take such as simply taking a few steps forward. Praise praise praise, treat treat treat :) If you're nervous he's going to mirror that as well, and since you are apparently pretty bonded with him it may be hard to tell at times if you're reacting/responding to his nerves or vice versa LOL.
Anyway, that's what I would do. Baby steps, tons of praise to build confidence and treats thrown in. Reprogramming the experience to undo the bad past. I am guessing that he came from an extremely abusive environment and that you're the one who's going to bring him out of it.