Pad or not, bareback is rougher on the horse, because all your weight is concentrated at the spot your butt sits. The pad just puts a little more padding under your ischial tuberosity (the two bones you actually "sit" on), but it will still feel the pressure at those two spots. For the occasional ride of relative short duration (an hour ride 2-3 times a week) you're unlikely to notice much. If you ride bareback on a regular bases (with or without a pad) your horse will end up with a sore back similar to what they get with a saddle that rocks (vs bridges). That's why saddle trees are used. To displace the weight of the rider and spread it over as wide an area as possible. It's the same reason why most large nations experimented with and abandoned using treeless saddles for their Cav. It was a wonderful dream to think of not having to get a properly fitting tree for every horse (no small task for a mounted military force). After the Hungarians and Germans came up with their new military saddle (what the English speaking world calls the "English" saddle) the Cav was much more versatile (that saddle allowed more freedom of motion so more could be done). This was great until they started having military campaigns. An unacceptably large number of horses were out of action (unfit for duty) during campaigns. Turns out that the saddle's tree design, while giving better maneuverability, did not displace weight as well so have a lot of continuous riding huge numbers of horses were out of action. Treeless saddles turned out to be even worse and never pass the testing phase since no tree left pretty much all horses ridden that much unusable. But I'm getting long winded and off track
Think of it this way.
I stand on your back with my heels. You feel the direct pressure and the muscle I'm on will eventually start to hurt.
I fold a blanket and put it on your back and do the same. You'll still feel the same points of pressure. It might not hurt as quickly, but it will hurt eventually.
I put a 24" square, 1/2" thick sheet of aluminum across your back and again stand on it with my heels. You won't feel the spot where my heels are and all the weight is spread out over all the points where the aluminum sheet is contacting your back. You'll be able to tolerate that a significant length of time more than the other two ways.
Most riders have ridden bareback from time to time. Long ago I'd done it to hurry up and get cows back in that had gotten out or if I want to ride down to take a swim in the river and didn't feel like taking the time to put on a saddle or if I wanted to ride my horse into the pond to swim her. It's unlikely to harm them when it's occasional and short vs regular and long.