First and foremost, you have to have 100% control over your horse's body parts before you ever go out on a trail. Secondly, you have to be 100% OK within yourself before you ask your horse to be 100% OK with what you are doing to it. Thirdly, you need to know the moment your horse loses focus on you and get that attention back immediately. I am guessing your horse "loses it" way before you notice anything is wrong. By the time you notice, he is already in over his head without any help from you. So, pay attention to his breathing. This is a huge indicator that he is getting uncomfortable. If he starts holding his breath stay where you are and ask for a circle or a leg yield, anything really, that gets the focus back on you and his breathing returns to normal. Then proceed. You might only make it one foot before he starts holding his breath. That's fine, just repeat the process.
And it isn't that they aren't scared or nervous going home, they just have all their focus on home, so less chance of noticing the scary bush or horse-eating log. Same thing, though. Don't let them rush, or pick the way. You make all the decisions for them. If it feels like they are anticipating a left turn, take them right.