Do not go alone!! Even if this is a legitimate job, I do not think that they would object to someone going with you, especially if it is several hours away. Tell him straight up that you plan on traveling with a friend and see if he backs off on his offer of a job. Your friend may need to stay in the car or keep a respectable distance while you are being interviewed, but you absolutely must
Not go alone. There are just too many weird people in the world.
If you get there and the wife and kids suddenly are out of town for a few days, get back in the car and leave immediately. You are going to be living there, the wife and kids would want to meet you. Big red flag.
Asking you to stay several days is a tricky one. I can understand him wanting to take some time to get to know you since it is a small operation and you will be living there, but unless I was staying in a hotel, I would not spend the night.
Asking for your physical stats is not uncommon if you are going to be riding. If you are just helping in the barn, he had no business asking. Not sure why he would want to know what you drive.
If the job is several hours away, why is he advertising so far from home? A live in job on a farm is usually not that hard to fill.
I would be all over the internet, searching his name, farm name, address, phone number, anything looking for clues that this is, or is not, for real. What type of operation is he running? Does he train? Sell? Breed? Then I would think that there would be ads, at least locally for his services. Check yellow pages for that area.
This may be a perfectly legit job, but your safety is the most important thing. If this guy is on the up and up, he should understand that.