Agree with others, a 1000 pound stock horse can hurt you just as badly and just as quickly as a 2000 pound draft. The most important thing will be that the horse is trained properly. Shoot, I am not that much bigger than you at 5'5" and 135 pounds but I have no problem riding and handling my drafties that are 17.2-18 hands and 2000+ pounds.
If you have a problem getting the halter on, then teach the horse to drop his head with a word or a tug on his mane. I had to do that with my Perch, John. Now, I can grab the end of either his forelock or an area of his mane just behind his poll and tug lightly and he'll drop his head to a manageable level.
As for mounting and saddling, it's more of a pain in the butt than a regular sized horse but so long as you keep a mounting block, step ladder, or a barrel around to stand on, you'll be just fine