If it weren't for 'older people', I certainly wouldn't have gotten the experience I did on the trails...I know it's not for everyone though, and I would rather ride with people who want to be with me, than someone doing it because they feel they don't have a choice.
To the OP, do what you feel is most comfortable for you and your horse; don't push him too far, especially on his first ride, if he seems nervous. If he is cool then go a little further. As has been mentioned your horse will only be as calm as you are, so ride confidently, and even if there are little spooks here and there, the ride should go pretty well.
"The ideal horseman has the courage of a lion, the patience of a saint, and the hands of a woman..."