Ya, that could easily be it, he's a big guy and carries himself big, which can be seriously intimidating. My horses are really only used to me handling them ever, and I'm not even quite 5 feet tall. He is much more of a dog person, kinda rough housing and goofy like, which my pony liked very much at first, he was a mouthy little testosterone filled pony.
After clicker training work the pony got over being so mouthy, but then like I said in the OP my fiance was approaching him, standing square and the pony took it as a challenge, kind of stood up to him. Then there was a noise out back so the pony turned away, my fiance moved forward. But the pony didn't notice, and when the pony turned back he flipped out, not expecting him to be so close.
My mare saw the events and when the pony flipped she did too.
I think this may have been the original cause of all this, then me leaving them alone the whole weekend certainly didn't help. If he wants I'll help him spend some time working on their clicker training, that should get them over him.
I guess I was just really surprised!
Thanks everyone for your responses :)