Doing the odd tiny jump(under 2') in a western saddle is okay. However doing multiple jumps on a regular basis over 2' and you are asking for problems. A western saddle does not allow the horse to bascule over the fence which means he/she can not use their backs properly. Also western saddles distribute weight over the horses back differently as well, which again does not let them use themselves.
Then as others have said, it doesn't allow you to go with the horse the way you should and speaking from experience, a horn in the gut over a fence hurts like an SOB lol.
As for cross country not being fun, well to each is own. But to me nothing beats the rush of you and your horse galloping full out to a drop or a water obstacle