I guess I should have clarified. I was not talking about felt or straw hats in terms of "expensive" hats and hairspray.I haven't had an issue with it on any of my hats, really nice felt and straw. They are all fine and have had it done a lot.
Again, you are talking about FELT ... not high quality beaver fur.As long as you use an unscented hair spray, it won't ruin the felt...I have used it on my felt hats for decades (on the outsied) so the felt will hold it's shape...still black as they were when I bought them 10+ years ago(well, after I clean them...).
Some associations have rules that require a hat be worn while you are competing, so it may not be feasible for the OP to just "not wear one".I say if you are not wearing a helmet, I wouldn't bother with the hat.