Horses are such creatures of habit. I usually give a horse, depending on it's nature, a couple days up to a week or two a chance to settle in a new environment before I start working them. I visit them in the new place as much as possible so they know I'm still there for them but the surroundings have changed.
I'll feed them there, hang out with them, pet and love them there, maybe lead them around a little, but no real work until they are sure that they are NOT going to be eaten by the scary horse monster.
Even switching pastures on the same land can be stressful for some higher strung horses.
Your saftey is always #1. I would make sure he's comfortable in his new surroundings and give him a chance to get used to the feeding schedule and then go to work with him.
If you think about it from his point of view, he was at the "new place" for 3 days...he was just starting to figure out that he was ok and going to get fed, and then he was bannished to the pasture just before dinner.....OH WHY, WHY CAN'T I STAY AND HAVE DINNER..... LOL... He was just panicking.
Give him a few days and try again. This is a big change for him.
Be strong and courageous, and do the work. Don't be afraid or discouraged by the size of the task, for the Lord God, my God, is with you. He will not fail you or forsake you.
1 Chronicles 28:20