Ground mounting....at 11 years of age and only off-track for 2 of those, he doesn't know what it is you are doing to him.
The horse is programmed to have a rider put up and off they go to condition and race...
The horse it sounds is accustomed to a working environment and relationship with their rider.
The animal is not accustomed to a outward show of affection from astride of you putting your arms around the neck to the horse...

You are off-balance and hanging on.......
Think about that and what to a well trained animal that means....rider going down!
I would suggest you save your bearhugs for from the ground for
your safety and the animal not capitalizing on you being very off-balance and dump you in a heartbeat.
Learn instead to give a nice neck rub, gentle pat and soft-kind word spoken...that to is praise and affection but far safer.
Save those bearhugs for happening from the ground.
I agree with Saranda though mounting from the ground.....not on this one and he be happy.
The prior owner may have been a very fluid, fast and spring in her step, or took a leg up and you just may not be any of those, or the horse didn't like ground mounting from them either.
When you are pulling yourself astride, all weight hanging from one side while you climb your way to sit the saddle...

The twisting of the horses muscles, nerves, vertebrae and skin....
it hurts.
So do right by the animal and learn to use a mounting block or similar
always, or get a leg up and see if the disagreeable attitude not disappear.
The animal
has issued its warning...take heed or take a kick.
Remember as a racer he was handled by professionals....
The majority of his life professional handlers on the ground and astride he had....
For 2 years it has been different, and he was not with his prior rider/handler for very long and now you have hands on.....the animal has a work ethic, not learned to be a pet of amateur's handling, yet.
He worked for a living.....big difference.
🐴....
jmo...