If your horse is giving you signals that he is not paying attention to you, and is nervous, you've got some work to do BEFORE you get on! Use this opportunity to solidify groundwork. Put the horse on the lunge line and work on transitions, walk - trot, walk - canter, canter -trot, and canter - walk. If he is attentive, and no longer spooky, then I might consider getting on the horse, if you feel he is behaving reliably. If not, put in a good groundwork session and call it a day! Life is too short, and there are so many opportunities to train, ride, and connect with your horse, there is no reason to put yourself at risk when you are getting signals that all is not well! Ride safe, and good luck!