There will be a doubtful nay-sayer in everything you do in life, regardless of how stupid it is for them to be negative about whatever it is they're negative about. I almost guarantee you have at least /one/ nay-sayer who doubts your cooking, dislikes your significant other, hates your car or where you live, says you shouldn't ride or drive or whatever else. Even if they don't say it to your face.
The thing about nay-sayers is simple. Don't pay them any attention, don't get wrapped up agreeing with them, accept their opinion for what it is and move on. You fell off, you got injured, you don't have any plans on changing what your hobby is (unless your going to sell the mare and quit riding for forever.)
Take your time in this recovery, do things right (getting a physical therapist or whatever,) so that you can enjoy the sport for even longer after you've healed. Listen to their opinions, discard them if you don't like what they have to say, and carry on. Because in the end it's your life. Don't let anyone negative control what you want to do.
And. Opinions are like armpits. Everyone has them, but they all produce a certain kind of stink.
(Yeah that's my kid friendly analogy for the popular saying.) ;)
The path is different for you and me, but the journey begins in the heart.