I suppose that if the ponies did not like doing their work, they would balk at it.
Ponies are very smart and have many ways of letting people know that they are so!
At my first riding school we had a lot of tourists in the summer, most were total novices. They horse's and ponies worked no more than 4 hours a day, even less really as that included mounting and dismounting. They trekked to the top of the Downs and back, the route did vary and more experienced riders did get taken off for a canter, the rest of the time it was just walking and trotting.
They did get bored, some more than others, so, those that were showing signs of objecting would remain at the stables after the day's work was done and we would take them down to the beach riding bareback. It meant a lot more work for them but
it was fun. They could charge around, spook at the waves, get down in the sand and roll.
Some would pretend to buck or would put in a big spook, behaviour they never did with a novice and we would make their day by falling off them or rather pretending to fall.
After such a session they were happy to work with the novices.
At a Riding for the Disabled school they were having problems with some of the animals not doing what they were asked. I could see that they were mostly totally fed up and suggested that the helpers, most were riders, took them out for a 'good jolly' charging about the countryside, once or twice a week. Problem solved.