He could just be stretching his back & neck out (the horses I ride do this to warm-up)- but if he's doing it the whole ride, you might not have enough contact with the bit, or you aren't using your legs & are relying on your reins.
But if it's just in the beginning of the ride, then that is normal, as he's stretching himself out. :)
I agree with the above quote 100%.
If your horse is stretching out and you have the rein on the mane, he is stretching himself out (for either just a stretch or for possible reasons given in another post...pain somewhere? Possible).
If he does this specifically when you pick up the rein, then you're relying too much on rein and not enough on your leg cues. In this case it's called rooting, and it's the horse's way of saying he's not found any relief/release in the pressure when he listens to the bit, so heck with it, he'll try and get his own release by rooting his nose forward and dropping his head to get away from the bit pressure.
If he's rooting then get into the habit of then never taking the slack out of the reins until you've put your legs on him to ask him to give you energy (legs are for energy, reins are to direct that energy). Even if it's just a little squeeze of your calves. The more he roots, the more leg you use to drive him into the bit and be more fair about using the reins....that is, only take out the slack and use pressure when you are really telling him something (slow down, give to the bit, turn, etc)....don't ride with your brakes on (constant pressure that means nothing)
As for the person that said a tie down...that's definitely not a good idea. With a horse that roots, it's like telling the horse "shut up and take the pressure" and that's just not being fair at all to the horse. Fix the problem (rider error of misuse of the reins, too much pressure lasting too long not telling the horse anything....riding with the brakes on....or pain.) and you get rid of the problem.