Grooming your horse is a bonding activity. Why don't you just take your time and groom him for a bit longer, instead of rugging him? If the weather is warm he really doesn't need that rug and for some horses it is uncomfortable to roll, to scratch, to be groomed by other horses, if they are rugged. To my mind, in too many cases rugs give comfort to humans - but the comfort of horses should go first. :)
(And, no, I am not a hyper anti-rug person. I have a fly sheet for riding (we have LOTS of bugs here) and a fleece rug for walking off in cold weather, but I just don't believe in using rugs if there isn't a medical condition.)
I have come a long way, to surrender my shadow to the shadow of my horse.