I am much like MHFQ, I am self employed so I have all the time in the world to spend at the barn.
I currently have 14 equine critters on my property, but many of them are retirees. In the big pasture, there are 8 of them, all of whom are either fully retired, semi retired, or not started under saddle yet (I have a coming 2 year old and a coming 3 year old). With them, I'll usually take 20-30 minutes a day and go scratch on everyone, hand out a few treats, and check them over.
In the smaller paddock, there are 4. 2 are a working driven team of drafts, one is a junk horse that I really want rid of, and one is our old retired Belgian Mule. The team gets worked an average of at least 1 time per week, Tiny (the mule) gets taken into the barn each day and given a special feed because he's starting to have trouble keeping his weight up, ******* (the junk horse) doesn't get handled much beyond basic care. That seems to suit him fine because he just doesn't like people very much.
The final 2 are mine and my Dad's personal riding horses. They are kept in a large pen beside the barn so that they are easily accessible and easily caught. I usually use Dobe (my guy) on a nearly daily basis. I will often use him to help me handle my young customer horses and/or take him for a relaxing plod around town in the evenings.
I would feel a lot worse about not spending much time with some of them if they were kept all by themselves. BUT, with them being pastured with a decent sized herd, they keep themselves socialized and happy. Sure, they are happy to see me when I go out there, but they don't seem to crave my attention like they would if they were being really neglected.