Awww I'm sorry you are having that experience! I think that you should always be open to trying new things with your horse, especially if you are both young and able! If your horse doesn't have the appropriate conformation it may not be able to go very far in dressage or jumping or eventing but that shouldn't stop you from trying it out and having fun!!!
I have a big, bulky horse and I am going to start him in dressage and see where we want to go from there. I have no show aspirations and no delusions that he will ever be a jumper or advance up through eventing ranks but by golly we are going to experiment and try things and enjoy life as best as we can!
My favorite thing to do when people tell me things like this is console myself by looking up youtube videos of palomino quarter horses doing amazing jumps and eventing and dressage. There is even a horse who is basically identical to him in color, markings and in huge stockiness who is a breathtaking eventer! Not that I expect to get that out of my guy but it gives me hope and inspiration to try new things.