I have a couple of young horses that I ride/train that go behind vertical when I start asking them to do hard work and they get tired. What has worked wonders for them is gently bumping them back up with both reins or just one, and riding them with the feeling of them being above the bit. Usually when I feel like they are slightly above the bit is when they are, in fact, just perfectly vertical. And then a good exercise is to leg yield in the trot, transition to walk and continue leg yield through the transition and vice versa.
I would not throw away the contact, since you want them to be through and powerful from behind into a nice light contact, thus making for a balanced, happy horse. So keep the contact, do minimal suppling when possible, and keep the conection soft through your elbows. And definitely reward your horse when he does the right thing! If you don't have mirrors to constantly check and see where he is vertical-wise, try to have a person watching you so you can keep tabs on it.
"He doth nothing but talk of his horses."