I thought I would hate it, but I actually find it more comfortable than a ball cap. I also ride through the bush a lot, and it protects my head from tree branches.
It's stretched out quite a bit over the last 3.5 years, so I can now wear a thin toque under it this winter. I'll have to buy a new one this spring, I guess.
I don't think I have anything to prove anymore as a rider, so I don't care if I look a little dorky with my western saddle and English helmet. Would hate to end up with a bad concussion or much worse if it can be avoided.
Just my thoughts. Everyone has their own risk tolerance level, and I can respect that.