The age at which horses stop noticeably gaining height can vary quite a lot - between 3 and 6 usually. However, Very few horses will stop growing completely before 6 years, even though the changes may be unnoticeable and at 4 years most horses will only just have their major joints maturing.
British Native breeds? You're looking at not being anything like mature until 4 yrs. I deal almost exclusively with them, and it's not unusual for them to gain a couple of inches or completely change in appearance as they mature between the ages of 4 and 6.
I think height-wise most ponies (except for Haflingers) will be mostly full grown around 4ish but will mature and fill out a bit past 4. I had a welsh/QH that had grown and filled out by 5. So much of it depends on the individual horse and their genetics.