If he wigged out over bit/rein pressure, or the trainer's leg on him, then he has some major holes in his training, that need to be taken care of. He was probably just very confused, as it sounds like the trainer just got on and rode off like he should "know" more than he obviously does.
Step back, and regroup in your training 'progress'...you may need to teach him to line drive, so he learns stopping and turning with the bit, before your daughter attempts to ride again. Desensitize his sides to various 'things' touching him. Sacks, plastic, etc...lower the stirrups so they can really move with him, and can touch him all along his sides.
I wouldn't quite give up on him yet...he's only 4...still very much a baby, so just take your time. If you can, look for another trainer in your area, and if you can, send him or have the trainer come out and work with him; glean as much as you can from the trainer, as well, so you have plenty of stuff to work on when the trainer is gone.
"The ideal horseman has the courage of a lion, the patience of a saint, and the hands of a woman..."