Saddle pads generally make matters worse when a saddle doesn't fit. It's like if you were to put on bigger socks for when you wear a pair of shoes that pinch. Riser pads are generally used for saddles that are too wide, but they weren't intended to be a miracle cure. Horses almost never fit "perfectly" in thier saddles, so the pads are there to make a decent fitting saddle fit better, kinda like a nice pair of insoles in your shoes.
Since it's western, poke around under the fleece behind the forks. Often there are screws, bolts, or nails there holding the saddle together. These things sometimes have a nasty habit of coming unthreaded or working themselves out when the saddle's older or seen a lot of use. You may need to have your fleece replaced. You might also double check and make sure the tree didn't somehow get broken.
For an older horse, you might like the treeless trail saddles. They make them in western, too. They're like glorified bareback pads. I have friends that bought them and don't ride in anything else since getting them. They look funny, but I hear they're wonderful!