Agree with all of the above. Remember, he's only 4yo, still immature & even if he was started early, he still hasn't had all that much experience. On top of that, new environment, new experiences - he's a prey animal, so rugs hanging on fences for eg aren't an 'even' or 'only' - they're one more potential threat. Horses are 'hyperspecific' and don't generalise well, so I'd be thinking about starting at the start with this young fella. On top of that, and very importantly, YOU are new to him. He needs to develop trust in you as a worthy leader, so he doesn't feel on his own in a scary world.
I'd personally stay at home or in an environment he's comfortable in - certainly not go play in the traffic
- until you develop a relationship with him. I'd also be inclined to then take him out & about on lead, or if poss. With another confident horse. I think depending on the level of his anxiety, Dawn's idea of working & resting *near* the 'scary' but not forcing him to approach it is a good one.
BUT depending on how edgy he is about it - maybe going out with you at all currently causes him stress which gradually escalates if it keeps going & the rug on the fence is just a 'final straw'. Because once a horse is frightened & emotionally reactive, even if you force him into something, he really CAN'T think clearly in that state & his fear can still escalate. So I'd ensure that you're not pushing boundaries too hard & when you're doing stressful stuff with him, keep it as low key as possible, with short sessions & stress free breaks in between.