I actually find a huge difference with wearing different boots while riding.
I find that my dress boots really stabilize my ankle from flopping out sideways, while soft boots or half chaps don't stabilize as much and in just paddock boots, or loose fitting boots I find my ankle to by quite floppy.
I have seen riders improve with the purchase of hard dress boots because the lower body becomes more stable.
However, with that said, at where you are now you wont feel much difference. But down the road, yes what boots you wear do make a difference!
They say money doesn't buy happiness -- well happiness doesn't buy horses!