ASAP.
For the reason Peppy stated - they can get injured early on! Then what? Does your foal's first experience of humans have to be discomfort? IMO, not if you can avoid it. IF the mare will let you without trying to murder you, I believe foals should at least be willing to wear a halter and walk next to a human by the time they are two or three weeks old. Some mares will not allow this - never put yourself in the situation of having something attached to the foal and you, and an angry mare who is not OK with it! People get killed that way.
A family friend bred a half-Arabian out of her mixed breed mare. He was basically unhandled when he tangled in a fence. He had the vet care he needed but healed with a very contracted tendon, which needed surgery to correct. He went on the trailer to a university-based vet hospital with his mother and was then forcibly weaned when Mom was taken back home that same day. He came home dangerous.
It was a long hard slog for his poor owner to care for his surgical wounds, and meanwhile she had to cope with his dangerous behaviour and is "too nice" - she didn't want to discipline him because he was hurting.
After he was healed he was even MORE dangerous because now he was feeling good and had learned he could get away with it! It's been difficult for his owner to learn to be more firm with him, but now he's not dangerous any more. We briefly had him on our property to teach him some more manners because his owner just physically did not have the strength or the endurance to up the ante when he did, but he went home about a month ago and so far all is well.
Anyway... moral of the story is, unhandled babies are a danger to themselves, and potentially become a danger to others!