This horse is not lost irrevocably: proof is that he now comes to YOU from pasture! That is a HUGE turnaround, & I congratulate you!
This horse, like all horses, doesn't care how much a person knows till he knows how much a person cares, & he knows that you care, simple as that.
Unless he finds the same heart-felt caring in another human, he's history; it couldn't be clearer. We all know the odds of that one aren't really good, unfortunately.
No one knows your financial/time available/et al situation like you do, so you're the best judge of whether you can handle keeping him!
There's no doubt in my mind, however, since he's come around to you so beautifully, that YOU would be able to get him safe & rideable. (not so sure about hubby; perhaps horse had a bad experience with a male(s)).
If you decide you're going to keep him, I suggest that you get Parelli Level 1 home study course, find out the horse's innate "horsenality"(the quivering in the corner means he's quite right-brain introvert at this time, it seems), & so first the groundwork, & only after he's got that down pat, would you think about asking his permission (yes, you should ask permission) to swing a leg over his back.
Let us know what you decide!