You are very lucky to be able to go see him as often as you like. I used to see my horse every day, but my life has changed a lot.. Now I have kids, had a break up, had to move to the city, and work a lot more, so I am lucky if I get out 2x a week.
So don't beat yourself up, I am sure the horse didn't mind, and will be happy to see you tomorrow. My advice is to enjoy the time you are able to spend now, as you never know what life will hit you with.
And yes, I think we all have passed on an opportunity at one point to go to the barn, then regretted it.
My latest was just yesterday. I always go to the barn Mondays after work because I don't have the kids that night. But I am selling my van, and had a prospective buyer coming. Well, I got stood up, and waited far too long. I shoulda left after a half hour and gone riding, instead I waited until it was too late, and was grumpy the rest of the night!