You have to be sure you are riding balance, side to side that the top of your hips are parallel to the ground. Many riders don't and compensate by dropping a shoulder. Instead of the hips and shoulders appearing like the letter I, it looks like > when someone looks at you from behind.
Check your balance and position. It could be possible you are collapsing your inside rib cage forcing your balance into your outside leg/stirrup. This can be an annoying habit and can cause mild aches.
Most riders who collapse on their inside rib-cage also do it when they are driving a car. It can be a difficult habit to break but once a rider overcomes it, it drastically improves their balance :)