Fairly typical behavior for all horses. When in the lead, they go slow. When behind they go fast because they want in front. When leaving the barn/trailer they go slow because the day is just starting. On the way back they hurry it up to get the ride over.
To achieve what you want it takes miles and miles of saddle blanket sweat and consistancy on your part.
-You leave and come back at exactly the same pace. Letting them rush back is what creates barn souring and people allow it to happen all the time. Reason why is all of the sudden your horse is eager to go and it's fun!
-When in front, keep him poked up at the speed you want to maintain. It's just as easy for us to become as complacent as the horse when in the lead, stay awake.
-When in back, work at keeping him at a decent gap behind the next horse. Yeah, I know it's not an easy task but it can be done and takes tons of patience.
Just remember, you give an inch just once and you've possibly undone days, weeks or months of work.