Yeah, that just sounds painful. You'd have to leave at a slow walk, not a trot! Besides, if you ride in a sports bra like I do, what you'd be leaving on the trail wouldn't look much more exciting than a tank top.
Yeah double up and leave it on the way back or stuff it in your pocket. I'd leave an older one I outgrew, I think I have my neon green and orange bra somewhere, no idea why I got it or why I kept it but now it will have a use!