I understand why many people want to breed their own mares, so I wont get into that, though I do believe, from personal experience, that you will have a much better chance of getting what you want if you buy.
That being said, if you are set on this guy, which it sounds like you are, don't bother asking us about it lol because as you have seen, many will advise you against it, mainly because he is grey.
First, determine what you wish to do with this foal and then decide. If you want it to have a show career, go with a different stud.
The fact that he can seemingly be out in a field and not breed mares could simply be because he is old and his hormones are no longer raging. Many studs mellow down with age, especially if they are kept in contact with other horses.
If he were kept like many studs his whole life (segregated completely and only able to interact with other horses when breeding) I can guarentee his behavior would be different.