If indeed he does have a retained testicle, he needs to have the surgery because he will always act like a stud and the problems that creates with everyday living.
We bought a beautiful 6 yr old Morgan gelding years ago, gelding being the key point. Because of his studdy behavior, we had a blood test done and he indeed had one testicle , he was gelded as a yearling but nobody said anything about only taking one testicle. After the blood test, we had the vet do the surgery, it was not particularly dangerous, he was done while in the stocks, standing and the incision was a small one in the flank. Testicle out, no more stud.... He did try to breed the mares, but never succeeded, his testicle was a "high flanker" and the sperm were not viable.
Get the blood test, if he does have a testicle, have it taken out for your safety and enjoyment of your horse and for him also.