I'm with the others - tell him to speak with your attorney at any time he tries to contact you. He is trying to intimidate you, and as the others have said I believe this was a scam from the getgo. I know hindsight is great, and you are kicking yourself for going through with it after the initial hesitation; but what is done is done and you can't change it. All you can do is follow the proper channels and trust that karma is going to kick this man's butt.
If you have done nothing wrong, and the vet has done nothing wrong, you have nothing to be afraid of. As was mentioned, keep record of EVERY text, email, call. Keep a journal of do not miss a beat. Make sure when you do go to court that you look for your legal fees.
I do wish you the best of luck, and I hope you can eventually come back and let us know that you sorted out this scheming toad.
Stop for a minute, open your mind, learn. You may not agree with what I say, I may not agree with what you say but we will both learn something new.