My 28 year old mare has a very slight, probably mostly age-related, sway [she has a long back and a weak loin connection - a sway was inevitable] and I can guarantee you that it makes her uncomfortable at times.
She can handle short bareback rides but anything longer than about 15 minutes and she starts fussing for her rider to get off.
Saddle fit is a real issue. Her sway is extremely mild, yet I haven't found a saddle that doesn't bridge. With a little extra padding, she's fine...but still. And after longer rides [2+hrs], her back is usually always mildly sensitive...so I try to avoid riding her for really long periods.
Thanks to her back, she also has a weight limit of about 150-160lbs. She just can't handle more than that.
Anyway, all I'm saying is that that sway IS going to catch up to this guy sooner rather than later. I got my girl at 23, only 7 years older than your guy, and all of that^ was as true then as it is now - and she had been fully retired since age 11!!
I love my mare more than anything and I would bring her home again, sway or no, but a sway isn't something to take lightly.
Here's a picture, so you can see jut how mild it is! Mild but so frustrating. Haha
Good luck. Personality-wise, he sounds wonderful!!