I can relate to what you are experiencing, LilacsBloom.
I started (at 36 years of age) on a big horse - or big for me anyway, being only 5.2 ft
I have a general problem with heights and I'm afraid of losing control and get thrown, so sitting on a taller horse can be an issue!
I was open about my fears with my instructor and I was later moved to a smaller horse (he is about 13.5 hh). I felt better with him, as he would "fit" better: my legs, for example, were long enough for him! LOL
I loved all the horses I've ridden but I can tell you this: when I moved to the pony, I knew what my instructor meant when she said "It's not their size, it's their temperament". I loved my pony to bits and would miss him when I could not attend the lesson, but boy, he was a good teacher too - where my big horses had been patient about all my mistakes, he would try and buck me off my seat if I wasn't getting it right after a couple of times
One day he also "took over" and I can definitely say ponies can be fast!
Once we got to know each other and I improved, we were like two peas in a pod! If I'd lose my balance, he would stop and wait for me to be fine again. I miss him so much!
But well, what I'm try to say is: I later realized that my instructor was right, big horses can be as gentle and as safe as small ones and it was a big mare who baptised me into riding and I owe her a lot.
So do not be afraid of bigger horses, they will give you lovely memories too and yes, groom your new horse because it helps bonding!
You also seem to have understanding instructors, so if you feel too uncomfortable and prefer the pony, just talk to them openly
I was never ashamed of riding a 13.7, actually I was quite proud of the two of us! I once said to a kid laughing at my pony: "If you want to ride the wind, you should try him!". That shut him up