I'll try to help you out with the things I have had experience with or I've been taught by my instructor/trainer. Also, I am a very big fan of Clinton Anderson, so some of the things I'll be saying are from him, as well.
I would start with desensitizing, myself -- it sounds like he could use a lot of that. Looking up Clinton Anderson's desensitizing videos is a great idea. He explains everything very clearly and to the point, and it works very well; I've done it. GiddyUpFlix.com: Horse DVD Rentals | Horse Training Dvd | Horse Dvds | Rent Horse Videos
is a wonderful place to rent any training videos. It'll help his confidence to not be worrying about everything he sees or feels. I would also install whips, sticks, empty milk bottles--anything he might be afraid of--from the rafters in his stall. You can also put an old basketball or something in there so he'll get used to having something around his feet.
I would probably start with yielding the hindquarters, because when you start working on sending him through spaces, you will need him to have that skill. That is part of the sending, really.
I did the flexing with my mare and it is well worth it! And it is one of the first things I'd start working on. You can have him do a few of those every now and then during a ride, and like you said, it's a great thing to do for a while after you mount up. It's also something you can go back to if he gets worried or doesn't want to stand still. So just say he won't stand still or even halt after you are already going: flex him until he stops his feet, and then immediately release. Flexing stops are actually a really good thing to practice in general, anyway.
First, before you work on him walking calmly, it is important that he is already walking calmly by your side when you are on the ground, and that he is respectful of your space. If he is, then you have the right idea about making him do pattern work. It is so much better to have them thinking about what YOU are wanting them to do, instead of about that super scary fence post, or whatever!