I used to do that sometimes back when I was a teenager. It probably wasn't the smartest idea, but I was a teenager.
For a lightweight person on a trustworthy horse, having a saddle or not doesn't really matter. When I rode with a rug, I would unhook the rear leg straps and hook them around the outside of the horse's legs (if that makes sense). They were useless but out of the way and not restrictive like that.
I will warn you that nylon is slippery. Riding a fresh horse in a nylon sheet on a cool windy day can be a bit interesting.