So don't feel bad! You need to do what is best for you! Ultimately, it is what is best for him, too. He needs someone who truly wants to be with him without a doubt, and you need to be with someone you know you want to be with!
Does that make any sense? It's been a long workday.