Sounds like your position isn't quite right. Start by establishing a solid 1/2 seat. At the halt or walk, stand straight up in your stirrups in balance to where you don't have to hang onto the horse for balance. Then without changing anything from the knee down, almost sit down in the saddle to where your hovering over it without actually touching it. This is the correct position. Practice holding this position at the walk, then trot, then canter until your thighs either burn and ignite into a fireball. Over time your legs will get stronger to where you can hold that position without hanging on for balance.
Once your half seat is good to go, practice it by trotting over ground poles making sure to do nothing except move JUST YOUR ARMS forward to release your horse's neck over the poles. Next move onto cross rails again concentrating on just releasing the horse's neck. It's important to establish the do nothing muscle memory or you'll end up in the jump ahead trap that most of us are guilty of. Good luck.
You just have to see your distance...you don't have to like it.