What can you desensitise him to? Lots of stuff is necessary if you're going to do much with him. Hooves, ropes, whip, saddle, trailer, you being behind him... the list goes on. It's the way that you do it that I think is vital. Perhaps he thinks he has to allow you to have his foot, but if he's trembling, he's definitely not happy about it & sounds like an explosion waiting to happen!
There are a number of ways to desensitise & I suggest you learn about 'approach & retreat' methods, to learn how you can do it with minimal confrontation & stress. For eg with his feet, I'd start wherever he's comfortable & only ask for a little more. Repeat this sort of thing, but in short 'sessions' so you don't overdo it. Only after he's comfortable with that step - not just accepting - would I ask him to accept a little more. I tend to be more gradual with a horse that's learned it must accept but is frightened about it than I'd do with a horse that just hasn't learned anything. It can be tedious to begin with, but it tends to get much quicker the more you do & rubs off on other 'tasks' as he learns to trust you.
So... With hoof eg, lets say he's now comfortable with you standing at his shoulder but not touching his leg. I'd start with him on a lead, not tied, so he can move his feet when he needs. I'd touch the top of his leg but only for a second or few(whatever you think he will take without panicking or shutting down). Then stop. Move away a bit. Start again. Get him comfortable with seconds of before minutes, at whatever step you're at. So say he's fine with having his legs touched & held but not picked up. When you first 'ask' with a bit of pressure & the instant he *starts* to pick it up, quit. Get him understanding about that cue before you hold his leg while it's up for a few seconds. Another reason it's important to do this gradually with feet is that every time the horse manages to get his foot away from you, especially in fear, the more he's likely to do it. But even little horses can be hard & dangerous to hang onto a hoof when they're reactive. So set it up to make it most likely for you *both* to win without any 'losers'. Make things as easy as possible for him. Ask for only small improvements at a time.