OK, this situation sucks. Let me tell you about something from the other side
In our barn, we had a situation where a young girl was talked into a rescue horse as her first horse. No one in the family had ever owned a horse. The kid, 12, had taken a couple dozen lessons. This horse was WAY out of her league. When an experienced rider was on this horse, she was fine. When the kid got on, with a death grip with her legs, the horse would absolutely freak out. She hit the dirt several times in the first few weeks.
This horse was basically just really sensitive. She was Saddlebred and high strung. She would react to less then half an ounce of pressure from leg or rein. To handle this horse, this kid needed years of lessons.
The day they hauled into the barn, the barn manager's husband, who had been a horse trader for several decades, told them to sell the horse. The barn manager took two or three days to express the same thought - this horse and kid combination was BAD.
Over the next two months, the parents quietly asked opinions of the other boarders, looking for a different opinion. They did not get one. Every single person flat said they needed to get rid of the horse before it killed thier daughter.
One day, my Saddlebred was havign an off day and took my daughter for a less than controled jigg across the barn. When the mother commented somethign about how odd that was, my daughter chimed in that no, it is just because the breed is so sensitive they occasionaly have mental meltdowns.
Instead of findng an appropiate horse for the kid, the parents went out and bought a 3D barrel horse for her, ALSO. So now the kid with no skills had two high strung horses to ride daily... They did not follow up on any offers of lessons or help.
At that point the stable owner asked them to move because the other boarders were actively avoiding going out when we knew they woudl be there. Since then, the first horse has died at the hands of another teen who tried to 'disapline' the horse out of being too sensitive by chasing her around an arena with her nose tied to the saddle. Her neck broke when she fell.
My opinion on your case. Go and be friendly, and talk to those who are makign the comments. Let them know how you feel, and get the whole story out of them. If you just need some help, keep pestering them to get it. You can do it and it will be well worth your while.
If everyone thinks you are way over your head... think about other options. Maybe send the horse out for training.