I also agree that you should keep him indoors if you don't want him wandering, but if you leave him outside, expect him to go wherever he pleases, and obviously if he's finding food elsewhere, either in a great hunting spot, or someone else putting food out for strays, then expect him to gravitate to that spot, and take turns at each place. I would go talk to the neighbors, and tell them your story, that you found the cat as a kitten ect., and let them know he is yours. I would definitely have him microchipped too though as an extra precaution. If you feel like the neighbors are trying to keep him, or have no respect for the fact that he is "yours", then I would keep him inside until they move if you are that attached to him. I hate to say it, but letting him live outdoors, you have to accept that he is a free cat. He will go where he wants to, and do what he wants to, and it is a very real possibility that he'll get killed by another animal, or seen as a stray, and someone may take him in.