Have you checked his saddle fit? My gelding stumbled all the time. He was purchased last year and I started riding then after years of being away from horses and having a bad horse accident just before purchasing him. His stumbling scared me.
His former owner said he never stumbled - he was a former race horse. If I took Red out up in the country where he was excited and really looking forward to going he didn't stumble very much - just a time or two over roots, holes, etc like any horse would.
He popped out with white marks on his back after going up to the state park with hilly terrain. I have tried other saddles since that time and he stopped stumbling. I think it was pinching a nerve. Check the saddle or pad fit and see if that helps. (I also had a number of farriers try to "fix" that problem and one farrier clipped him too short - ugh that was another nasty problem!)