I say do it, it doesn't mean you don't love the girls you ride with or your trainer any less. You need to progress, not just yourself but your horse as well. I know most horses love a new challenge, love to learn. If you never leave you will never know how far you can go.
You can still catch up with your friends, you can still drop by to see your trainer. Sometimes we have to make hard decisions but the results are rewarding.
You will come across this in life again, changing jobs, moving towns, etc etc. This is one step that you will have to redo in life again and again. Simply with a different situation each time.