When I start a horse back to work (one that has had a real long time off), I start them for about an hour in my fields which has some hills to it. I'll do a lot of walking for the first week (4 days) and introduce trotting by the end of the week. On the second week I'll take them out on a trail for an hour or 2 at the most. On the trail I try to follow my method of doing a Hunter's Pace - walk down hills, trot the flats, canter up hill. After about a month, I feel my horse is ready for a typical 2 to 4 hour ride but I still take it easy for a while and monitor her. BTW, my horses are on 24/7 turnout.
It sounds to me like you are not pushing your horse enough.