In my opinion, this horse sounds like he needs A LOT of work from the ground up and needs to learn the very basics. If he's a rescue horse, and you don't know his background ... from what you're saying, he sounds very very green. Not trying to sound mean or anything, but I think you should see about getting a trainer to help you work with him. If you don't know how to be correcting these things (and believe me, I wouldn't either) then a trainer who can work one on one with him, building him back up is what he needs. That's great though that you have a rescue horse. I just don't know if right now it's entirely safe for you to be working with this horse. You don't know where he's from or anything. You said he has great manners otherwise right? That's great to hear.
But I would definitely suggest getting someone more experienced with handling green horses to work with you, or maybe even do a half-lease. Again, not trying to sound mean, but I really think some basic but disciplined training from an expert is what this horse needs; some sort of regime to get him on track. Good luck though and let us know how to it works out. :)