Yeah, I'd pass. The simple fact that the vet doesn't know why the horse is losing weight sends up red flags. That horse is going to cost a crazy amount in feed and even then he might not gain right. It might be something as simple as worms or needing his teeth checked, but then again it might be something serious that's going to cost a lot in the long run.
The sway back, in my opinion is not as big a deal. If you were just doing light trail ride, then he would probably be ok. But still, unless the horse just had an exceptionally good personality or something, in consideration that your a newbie, I would pass. Keep looking, and find someone that knows horses to help you look.
There is no sport like barrel racing. It requires more speed than a sprinter, more balance than a gymnast, more grace than a dancer, and a stronger team than football.