I don't know if this has been mentioned already (just ignore if it has), but you need to be keeping constant contact with the horse's mouth. That doesn't mean 'constantly pull back', it just means that you need to be feeling his mouth at all times.
Second thing, which I saw had been mentioned, was the fact that you're constantly making contact with his stomach. This is a bad habit. If your horse doesn't respond after contact, let him know that he should be responding (a crop or firmer contact will do the job). As is, you're teaching him to ignore your contact. ^^