The first thing you need to do is get him to accept you, and accept you touching him safely. Even if he has been handled, you can't just walk up to a strange horse and start brushing his tail!
I'd suggest you spend a lot of time just sitting near him. Bring a book and a folding chair, and sit where he can see you. Do this daily if you can, for a couple of weeks. Doesn't have to be for very long. Stay safe - you may want to start by setting up outside a fence, then inside it, but with your back against the fence. If he shows signs of aggression, get out. You may want to carry a crop or a small tree branch to send him away if necessary. Once he accepts you (he may start grazing near you for company, come over and sniff you, or just ignore you), you can start walking up to him and just letting him sniff you. You might offer treats - or not. Watch his body language as you walk up to him. If he turns away or moves away, stop and wait. Don't push it. When he looks at you, move towards him slowly (never from behind of course). When you're about 3 feet away, offer your hand. This can be done even if he's used to being handled because he will learn to trust you. Once he sniffs your hand or accepts the treat, move away again. Do this for a couple more weeks. This will teach him that you're not going to make him work or halter him every time you approach him. He may start to follow you around. That's good, but ok if he doesn't. Then you can start to "groom" him. Start with scratches at the shoulder, the withers, the neck. Do this regularly too. Move around his body, being mindful of what he likes and what he doesn't. If he moves away from you, let him. Some horses like scratches more than others, but most enjoy it in some spots. This should allow you to get him used to you touching him all over, so that eventually, he can be groomed properly. All of this is inspired by a program called Horsefulness Training which I am doing with my spooky mare. She has gone from a horse that always runs away from me, to a horse that seeks out my company - always completely at liberty. With a horse that has been abused, you need to work on trust before anything else.
Of course you need permission from the owners to do any of this, but it sounds like they're already ok with it.