How much opportunity does he have to roll around in the sand, snow, grass, whatever? That does a far better job of cleaning down deep then what the average grooming job does. If he isn't itchy or sore there, I would continue to brush him out, looking for any sign of irritation. If he did get something from the pad and it's never had a chance to be properly cleaned out, then the problem remains, right? Look for flakes, lice, mites, etc. Though why a bug wouldn't migrate is beyond me.
Also, it depends a bit on the fur coat that your horse has at the moment. I noticed that my horse's fur is starting to be coarser in preparation for shedding. This time of year I can never get her to look good.
When you do start riding again, make sure your pad is washed, well-rinsed and dried before using it again. Consider what kind of pad you use - perhaps he is sensitive to that fabric/fleece? After a ride, brush him down with a wet brush to help get the salt/sweat off.