It takes me like 15 mins a day to take care of my horses and I have two living at my parent's house.
Pasture maintenance does take some time, as does fence repair, hay storage, and all the little chores you can find, but no where near 2-4 hours a day. Beyond pasture maintenance and heated water trough during the winter (approx $100 a month From Nov-Mar), my two cost me (or my parents) next to nothing. The water trough expense is going away as well because I built a solar powered water trough that works pretty darn well.
There will be some trial and error, talk to various people to find out what their time savers are before you start, both people that do it like you want to and others. If you're like me, work on setting things up to be the easiest/quickest care possible while still being as healthy for the horses as you can make it.
ETA - My parents do very little with my horses and all by their own choice too, so this isn't a case of someone sloughing off work on someone else. I find it very easy for the most part and little "work" to keep my horses.