I guess there really is no right or wrong.. I've had the same thought as some of you. Horses often travel from one place to another to show, get off, and get right to work. There's not much of a difference for him traveling 4 hours vs. going to a local show.
I think I may let him rest a little bit when he arrives. I'll walk him around, and may just rinse him off/lightly groom him if it's really hot out. I'll continue to do very light work with him, and maybe do a little lunging on that Sunday. As much as I'm dying to just jump in and ride, I want him to be comfortable and relaxed.