Recently I started doubting my ability because I was always riding a mare that I simply couldn't get to focus and concentrate, and Alli who has only been backed for about a month and a half now. I didn't even realise untill I rode a riding school horse who was one of the "most difficult on the yard" and he did everything perfectly for me :) major confidence boost. Next time I rode Alli, Claire (instructor/friend/stablemate) commented on how confident I looked on Alli.
Sorry I rambled but what I'm saying is go get yourself on another pony who lets you concentrate on you and how you ride. Get back on the horse your having issues with and ride with the confidence you gained from the other pony