Signs of a Beaten Horse

I’m trying to figure out what is wrong with a pony. He isn’t mine, he belongs to a family friend. He was pretty underweight when they got him. At first, after getting him to a healthy weight, he was a good little pony. He was sold as a kids pony. But now he is bucking and is very hard to catch. I’m working with him in a round pen right now and if I walk up to him he is extremely jumpy and literally shivers if I put my hand out. He is very protective of his right side. If I put my hand too high he freaks out. Not like a normal spook but eyes wide and shivering. Hates being patted. Once the halter is on he seems like he’s shut down. He seems zoned out. I think who ever broke him just threw a saddle on and rode him. Skipped everything else.
I’m not saying he was abused but it seems pretty likely, to me. What do y’all think?