|I spend most of my time with him working him and punishing him for being disrespectful|
I highly recomend getting clinton andersons book, or even buck brannamans dvds. They will give you all the groundwork to make your horse a respectful follower.
I actually tend to gravitate towards the dominant type horses, which is what my arab is now. Once you build trust and respect(which is what your gelding is lacking) you wont have any problems riding out with him alone.
I know many people in the whole, "be gentle, never hit, disapline the horse or raise your voice", but the truth is, that kind of behavior is forein to the horse. Watch them in the pasture. The lead horse reinforces his possition by any means nessesary, until the herd members learn to respect him as leader. First he might pin his ears, then glare at the offender and move threateningly. ONLY after the warnings does he kick or bite. Sometimes humans need to reinforce their possition as leader firmly.