I think it's very easy to feel that way, and I'm sure everyone on here has at one time or another! You can't get discouraged or frustrated, esp when things aren't how you wish they were. I honestly feel like the more you learn about horses the more you realize there is to learn! I think that's the beauty/challenge of horses, no matter how great you are with them, there's always room to grow. Be patient, but proud of how much you've accomplished so far!
As far as Blu 'wanting' to be with you, maybe he wants to be with you but just doesn't show it the way you want him to.