Here are some random thoughts.
1. Using the whip to thwap him every now and then won't desensitize him to your leg. If anything, it could make him more responsive, if he understands that that will be what happens if he fails to listen to your leg. I believe, and better riders can correct me, that you only desensitive horses to similar aids. In other words, using the whip to thwap him all the time would desensitize him to tap-tap-tapping with the whip, and using lots of leg pressure all the time will desensitize him to gentler leg pressure.
2. Having said that, I don't like the idea of you (a young, beginning rider) trying to thwap this horse and then see what happens. A lot of horses buck when you do this, and as an ex-racehorse he might just take off on you. If you must do it, make sure you're wearing a helmet and on soft ground, but a small enough area where he can't run away with you.
3. Do you have an instructor available? If not, could you have one of your brothers watch you ride him, then watch the other brother ride him, and see if you're doing anything different?
4. When you say he's a stallion, do you mean an actual stallion, or is his a gelding? If a stallion, try to get him gelded ASAP. People do ride stallions, but generally beginners do not.
5. Do you have access to an experienced female rider? She could ride him and see if it's a gender thing. If he's only ever been raced, maybe he's never been ridden by a female before and maybe, especially if he is a stallion, he doesn't think he needs to listen to a female. If another female rides him and he behaves for her, then you know it's definitely you.