Some years ago when I was at 'home' having a holiday I took my two nieces aged 5.5 and 4 years to visit a friend. He asked me to come down with him to see some young horses - these were mostly Shire X TBs so big.
I never realised that he had so many because there were about 30 of these youngsters in a large field.
The horses came over and were milling around. Nothing bad mannered, just more curious over the little people.
I picked up my youngest niece as she was very weak from chemotherapy she was undergoing and had the elder right by my side.
One curious three year old walked a little close and I pushed him back. The owner asked me what I thought of another and I turned to look. The horse walked forward into my eldest niece, she put a hand on his nose and told him back, when he didn't obey and continued to push she gave him such a hard open handed whack across the flat of his muzzle that it jumped backwards.
The owner laughed and praised her saying "That's right, you teach him some manners!"
I would never have taken the girls into that field had I realised just how many horses were there and I think you are silly and taking unnecessary risks by allowing them around a young colt.
Colts think life is a big party and can also be very handy with front feet as well as teeth.