it sounds like this gelding does not like it when you face him, does not really enjoy being touched . He is not convinced that being with you is the best place to be.
that's what i'd be working on. in the round pen, and out taking him for walks. the nice thing about walks is that you are in front ofhim, where he really doesn't want you to be, but you are moving away from him, and this is often much more comfortable for a horse. it can really build confidence in a horse to go for a long walk with you. just even have him go with you while you work on fences, or dig holes or pull weeds or anything. his job is to jsut hang with you.
in the round pen, rather than working on having him run around, and using the whip or your body pressure to 'make' him move his hind end around, just work on getting him to move away from you , toward the outside edge, but then , without necessarily running him around, see if you can't engage his interest in coming back to you. over and over again, you send him away (easy, easy) and draw him back. he will eventually leave you slowly and relaxed because he knows that he need not flip from being near you to getting away (fleeing).
a lot of horses if you send them away, like from the center of the round pen out to the edge, so you can move them in circles, will flip over from a mental mode where they are trying to find emotional rest next to you, to being in mental anxiety about the turning away from you, so they flee/rush off. in that moment, you have lost them, and anxiety takes over. if you can get him to be ok about moving away, and ok about coming back, I think he'll feel more secure in general about being with you.
a timid horse doesn't benefit from having someone 'chasing' his hindquarter around to make him stop and turn. he will only become more anxious. for turns, draw his head and mind to you, rather than pushing his hind end away from you. the result is similar in that the horse ends up facing you, but how he FEELS about it is totally different.
I also agree that bringing him in for a bit of grain will make him more happy about coming in.