My project pony this year was goofy like that; no patience, and no ground manners, especially.
Just be persistent, and patient with him, and don't let him get away with things; by all means don't abuse him...duh! But like you already stated, he needs a good dose of good solid ground work, and reminding of what manners are. Once those are under way, you will have an easier time of working him undersaddle, because he will understand that you are his leader, and that he can trust you.
"The ideal horseman has the courage of a lion, the patience of a saint, and the hands of a woman..."