You don't have to ride. Plenty of people just keep them as pets, including me. My horses can be ridden, but I'd rather just hang out with them. :3
Trust doesn't come free. You haven't known your horse long enough to be able to read him. You probably don't know how he reacts to different situations, what he'll tolerate, etc. The only way to know him is to spend lots of time with him. If you work with him constantly, you'll know him inside and out and you'll be much more confident in him (and he in you). Give him a chance. He sounds like a nice horse.