Really you should always be sitting, standing and walking with a straight back even when not riding. If you've been riding for a long time it can be easier to get a straight back.
I'd say I have very good posture due to drumming as well as horse riding.
Shoulds scrunched up, put back then relax them and put them down.
Up, back, down.
Make sure you're holding the reins at a good length as well. It helps.
Position yourself on the horse, straighten your back and get someone to put a riding crop through the gaps in your arms passing your back. If you slump your back simply the crop will break or bend.
I've found that thinking about "lifting your ribcage" is a good way to remember! Strengthening your core is also really helpful! Sit up, with your shoulders rolled back and hold yourself steady in your tight core.