I say go for it!
Most horses make the switch without batting an eye, as long as the saddle is properly fitted and adjusted. As KV said, that can be tricky, and your best bet is to have a professional saddle fitter (or someone non-pro, a friend or something, who has a lot of experience with fitting English saddles) evaluate the fit. If that isn't possible for whatever reason, do as much research on signs of good and poor fit as possible.
I switched my first horse from western to English without a problem - the biggest issue was getting him workable in a snaffle after a lifetime in an ATT, but it worked out without any drama. My current horse I bought with no history whatever, probably trained as you described and then left to stand in the pasture. Tossed the English saddle on, and haven't had any problems that any other green horse doesn't have.
If you are just fiddling around and experimenting, you can just go for it. There's a lot to be learned from this board alone to guide you. If you're looking to get into showing, or even just learn the finer points and really jump start your progress, I highly recommend taking at least a few lessons with a good instructor to help you out. If you want to jump, definitely seek some pro instruction before attempting that, for the safety of all involved. Jumping just has that extra tricky element.
Good luck, and have fun!!