If you feel like riding is not fun or is stressful, I would back off of it for now. You can do a LOT to build a good relationship with him doing ground work and just generally spending time with him.
After I had to sell my mare due to the economy, I had no desire to ride even though I had free access to one of the best horses I have ever ridden. Prior to this, I would beg my friend to let me ride her horse because I just loved how fine tuned and well trained he was. My mare had been lame for almost 2 years before I sold her, so I had not even ridden her all that much, but as soon as she was gone, so was my desire to ride at all. My best friend would rearrange her schedule and coax me to come riding with her, but it was always a labor not a joy. We both realized this and stopped trying to force it and after about 8 months, I just got the itch out of the blue. I started riding with her every chance I got and even stepped up and bought a new filly to train. Now, even though she lives almost 2 hours away, I make the drive an average of once a week to ride with her while my little gal grows up.
My advice is to not put pressure on yourself. One day, you will just know the answer, but don't force it. Chances are you are in a rut and if you just relax and not worry about it, you will get the fire back. :)