Firstly, agree with Fingerlake's post re desensitising to scary things generally.
Originally Posted by liftoffdolly
handle even worst he has no respect for personal space in the first place so will quite happily knock us over without realising don't get the wrong idea he's not mean just silly we've had him
With no info on how you handle him, can only give general stuff too. Don't blame him for it, perhaps he's never been taught human 'manners' before. With the pigs, he's not being a 'Prat' he's being frightened & reactive in a 'dangerous'(how's he to know it's not?) situation.
I'd be teaching him to 'respect' my space & yielding to pressure(responding softly, not reacting) in all manner of ways and basically getting things together on the ground & in familiar, low stress environment, before taking him out, before riding. Done well, this will help his confidence & trust in you, which will carry over to make him safer under saddle, because he can trust you to look out for him, rather than finding himself in a 'dangerous' situation with no one to turn to.