You are doing the right thing in carrying a crop under your arm and ready to go at all times. Be constantly ready to react instantly, not 5 or 10 seconds after the bad behavior. Stand up very straight, shoulders back and walk with great purpose and pretend in your body and in your mind both that you are 10 feet tall and completely in charge. Learn to cultivated a deep, gruff voice when you are unhappy with his behavior-no little girl squeaks!
Go after him hard and back him off but stop the instant his bad behavior stops. A smack to a horse that is doing nothing wrong can do more harm than good. Now, if your mother is involved in feeding this horse, she must learn to do the exact same thing- no wimpyness allowed! Either that or she should stay away because her being soft can undo everything you accomplish.
Please understand that in the mind of any horse, this will NOT make him "not like you". Horses simply don't think that way unless they are horribly abused over a long period of time. YOU ARE, AND MUST REMAIN THE BOSS MARE at all times and even if you are nervous, become an actress and never let them know it. Carry your crop under your arm or in your boot and watch him-spending time just watching and analyzing horses can be a huge education in how they think.