You say your horse yields fine on the ground when you "poke" him, but does he yield when you simply apply pressure on the ground? Also, are you applying the pressure where you legs will be when riding?
Whether a horse yields to lateral leg pressure when mounted has a lot to do with balance -- balance of the horse, but also, balance of the rider. Many riders don't realize how their body positioning is inhibiting their horse from doing what they are asking it to do.
For example, I have seen a video of a well-known trainer talking about "opening the door" by pulling his leg away from the horse to encourage the horse to move in that direction. This trainer was admittedly exaggerating this, but it still didn't appear that he realized what else was happening. As he drew his right leg away from the horse, he leaned his upper body to the left. Although the horse yielded its front end to the right, it was also compelled to move its hind to the left to maintain balance under the rider.
A more subtle example was a young woman who called me in to help her get her horse to side pass better. She could get the mare to move in one direction but not the other. When I first observed her, I could see the problem but could not discern the cause. I mounted the horse to see if I could figure out the cause and find a solution. The horse moved in both directions easily for me. This told me the problem was with the rider, so I had her remount. This time I looked more closely and discovered that her right shoulder was about an inch lower than her left. She knew her body well enough that she could correct this with my feedback. Her horse moved easily in both directions.
Balance is critical if you don't want to fight against yourself and your horse as many people end up doing. Try to feel how you are balanced. Tell someone on the ground what to look for an give you feedback. Take videos of yourself and your efforts; watch to see how you are sitting and how you may make it easier for your horse to both understand what you want and comply with your wishes.