Im not an expert, and mayby Im doing it wrong, but I always ride my horse like normal, and nothig has happened to him. I have only owned a horse for a few years but nobody has ever told me differently. I can't see it hurting them, but You may want to get other peoples opinions though.
Try reading the dewrmer box, mayby it will give you fuller instructions?
If its hot out its not good to ride them same day as worming...something about over heating and wormer causing colic or the sense...i don't really remember...but I've always just heard that its bad on the same day as worming and only if its hot enough to make them sweat while riding.
By the day after the wormer has just about been digested so it shouldnt matter if you ride him today. Plus its only spring. It will be hot enough for a horse to sweat but not to over heat.
Some horses get a little colicky after worming so maybe take it easy on him. You should ride for sure but warm him up well and stick to short canters. If you have a very regular worming routine (which should include fecal counts!) there shouldn't be a risk for impaction.
I have never considered de-worming in my riding schedule, but I always give it at feeding time and I never ride right before or after feeding. Certainly a day later is not an issue. With a horse that has any health issues, I would reconsider; if all is normal and healthy -- go!