I agree with Dancing Arabian that you should get some outside help with him.
You say he is 6 years old and not going to change but in my mind that is still a young horse with a lot to learn and there's no reason why with lots of training and groundwork that he shouldnt settle down and become the trustworthy mount that you want him to be.
There are things you can do yourself to help him too - setting up 'confidence courses' on the ground with strange objects lying around will help him get used to new things. Don't ride him, just lead him around these things getting slowly closer until he takes no notice of them. I have done this with much success, I tied plastic bags to the fences, put ropes on the ground for him to step over. Pretty much anything lying around can be used - just ignore him if he plays up and keep leading him past them. Anything that flaps or makes noise, I even tied an open umbrella to a tree - you can get pretty creative!
He may be feeding off your nerves too, if you are tense and waiting for him to spook then he probably will. Definitely get some outside help before writing him off though, it sounds like he is very special to you and you could really regret it down the line if you sell.
Don't push yourself to do anything you arent comfortable with though, your safety is the main thing. Take it slow and start from the ground with him.