nawww he sounds very sweet :) good idea about not rushing, it always takes time with a new pony. If you were only to bring in one other pony I would more lean towards a gelding (personal choice, obviously you don't have too) and have the gelding on trial to see how things work out.
The only reason I would go a gelding is so then they can form a bit of a 'bro pack' even though he was not bred, it doesn't mean that he would 'feel his oats' if he is put in with a mare, but on the other hand if you leave it for a bit he may be perfectly fine.
Was there any reason he was left by himself as a stallion? As in, did the owners say he was mean to other geldings, or was that just the way he was kept at that property?