Whether the owner comes earlier or later is irrelevant. Neglect is neglect is neglect. Whether they come at all or come all the time. If they are sick or disabled, either they think the horse is being cared for and would WANT to know, or they don't care and it needs to be taken away.
Maybe it's just my barn, but I know my barn owner doesn't tolerate anyone lacking on care for the horses - the boarder's horses are cared for to an at least acceptable level, or they are asked to leave and she notifies AC.