Back before smart phones, I'd stick my flip phone in my back pocket of my jeans, but it would always worm its way up out of the pocket constantly and I'd have to shove it back down. Occasionally I wouldn't notice and it would fall to the ground. Then one day, this happened while trotting and the horse stepped on it and killed it :P
These days, I almost always leave it in the car, even in the blazing heat or the freezing winters, and I've never had a phone harmed in years of doing this. Never really thought about the heat hurting it... I'd rather not set it somewhere in the barn or on the fence because it's too likely a person, horse, or cat will accidentally knock it off and damage it (like what happened to a friend's phone left on a bench - cracked her screen). My horse LOVES to knock things off of the fence like water bottles, so it would be very risky to stick my phone on top of the fence with him around.
I will zip it into my coat pocket in the winter if I'm riding alone sometimes, but I have no such pockets in the summer. My horse is a pretty spooky guy, and my phone has rung while in my coat pocket a few times, but thankfully I guess that's not one of the things he's spooky about :P