Welcome to the forum!
Sometimes a little reverse psychology can help. If you have the space, when he starts trotting, still your seat a little and bring him back down for a stride or two (just so its your idea
), and put him into a forward working trot and start doing some loops and figure 8's in the general area. When he starts offering to slow down, push him a few strides more (again, just so its your idea
) and transition down and reward him. Let him think that trotting is just such a good idea that you're going to steal it from him and you're going to do a lot of it!
It isn't too helpful if you're in dense woods, but in a fair sized yard or field it's done wonders for me. If you have access to an arena or a fenced yard with safe footing, maybe try riding him in there alone to see if its a matter of being alone, or being outside alone.
Also, check yourself to be sure that you aren't tensing up more when you ride out alone. My first horse was very attuned to that kind of thing - if I was at all tense, he was a jiggy, nervous head case. Sit deep, and sit the way you want him to move - slow, steady, in 4 beat rhythm.