^^^ This, is my thought alsoIf the new barn is just across the road he can still smell his friends and would rather be with them. This would be my guess about why he is stressing.
He knows 'home and buddies are just across the street,and would rather be there
You could try , as test, to put your mares across from him in the new place, and I bet he would settle down
It is one thing to take your horse to some different barn miles away, where he no longer knows where home and buddies are, and to have him close like that
Another scenario that shows this facet, is many horses that are a problem riding out from home, wanting to rush back, ect, do fine hauled somewhere on atrail ride, without that 'magnet' close by