It is exactly as everyone on this thread has said, but the most important one to me is DO NOT go back when the horse gets spooked. You are teaching it that whenever it wants to go home, spooking will get him his way. You need to end the ride when you decide it should be ended - preferably when he is calm and collected. And to keep your horse calm, remind him you are there - talk to him, pet his neck, sing, stop occasionally to give him some love. Go a bit farther each day, and you'll be on the trail in no time.
When we first bought our horses, two big draft crosses, they were great on the trails... As long as we were with other horses. When we tried going ourselves, we got across the street (still in barn view) and stopped. Wouldn't move for anything - not a smack in the but, not any amount of kicking, not spinning them in circles then going down the trail. We gave up and went back - a victory for the horses. But the next day, we put bridles over halters and when they stopped, we got off, clipped the lead ropes to the halters and dragged them - quite the feat, dragging two one ton horses. There was much butt smacking. But once on the trail and out of sight of the barn, they were fine. We had to walk them out there for quite a while, but it did eventually get easier until we could ride out there - but that first day, it took us nearly 45 minutes to drag them out (I will admit, there was some pulling on the reins as well, but now I wouldn't do that).
The moral of the story: Even if you have to drag a scared, stubborn mule past his point of turning back, do it. It only gets easier after the first time.