About the saddle, I'm pretty sure it's a few inches too small for you which probably isn't helping you be effective or him be comfortable. :)
If it's any consolation to you, if he threw his last owner, his last owner must have been a terrible rider or must have done something really stupid because most of the horses I know, if they were looking as uncomfortable as he was right there, would have tried to throw you right there. And since he didn't even really try anything funny, just looked super uncomfortable, I'd bet that you'll have to work hard to get him to throw you. :)
*hugs* I totally understand how you're feeling. I still remember the fourth day I had Lacey (back when she was certifiably psycho) when this dumb girl decided to tack her up and ride her without lunging her first and without asking me. I discovered them right as the girl climbed on and Lacey bolted, bucking as hard as she could until the girl came off. After that I was terrified of riding her. Lacey would jump around a little and I would get right off. Eventually, I realized that it really wasn't Lacey's fault and that I needed to trust her. Yknow what happened once I started trusting her? She started calming right down. Now I make it a point of doing things that give me less control with her sometimes just to show myself "look, you have a good horse, she could take off right now and there would be no stopping her, but she's turning off just your leg even though she desperately wants to go the other direction." Currently, I'm the only person that's ridden Lacey more than once that hasn't fallen off her, and I've ridden her more than anyone else in the last 2 years! So, have a little faith in Cinny, and yourself. I think once you loosen up and bring your energy down, you two will be just fine. :)
Another thought: if he neckreins, have you tried riding him in a western saddle? I tried to make Lacey go english for the first year and a half, but one day I decided, out of the blue, to tack her up in a western saddle. She was trained as a dressage horse, way back in the day, so I'm pretty sure she's never had extensive time in a western saddle. Well, she took to the western saddle like it was her new best friend. She started getting calmer and moving better, it was just an all around change. Personally, I still prefer english but since Lacey doesn't, I'm going to go with what she wants since it's not just a little preference, she really obviously prefers western.
Good luck! :)
Fabio - 13 year old Arabian/Lipizzaner gelding
Hazel - 14 year old Angora goat
Atticus - 4 year old LaMancha/Alpine cross goat
Rest peacefully, Lacey.