First of all -- horses make terrible 'pets'. Teach your little brothers that horses are not pets -- they are livestock.
Then, teach this horse respect and good manners. I have seldom seen a horse single out any one or any kind of person to disrespect if the horse has good manners and respect for some of their handlers.
Children should not interact with any horse unless the horse is well mannered and under the control of an adult. Then, the adult can scold the horse for even a 'cross look' aimed at any person, horse, dog or ????
I never let children go out in a pasture or paddock where there are loose horses. It is just too easy for a child to get kicked or run over as loose horse move around and interact with each other. Obviously, children are not good at 'reading' horses or their intentions. It is just too easy for one to be in the wrong place at the wrong time.
Also remember, if a person runs or walks away from a horse that has threatened them or scared them, they are 'teaching' that horse to be aggressive. Never let them be in that position. They are not old enough or skilled enough to 'hold their ground' and make the horse move, so never put them in that position unless you want a mean horse.
A few times I have seen horses single out small kids when small kids have thrown things at them or harassed them. I have seen a couple of horses lay ears back and run at children only to find out that children had poked them through a fence with sticks or thrown things at them. So, make sure the children have been taught to never tease a horse (or dog or any animal).
If your little brothers want to learn how to handle horses, teach them the correct way to lead and brush a horse. My little granddaughters (3 and 4 1/2) have been helping me lead the trail horses out to pasture or to the tack room since they were 2. They think it is a really big deal to help Grandma bring in horses.
If you have a young horse, you have an obligation to get serious about teaching it correctly from the beginning. Otherwise, you will create a monster that ends up being dumped at some sale or abandoned out in some back pasture.