Re right of way: You'd have to check your local municipality/county to see what they have in place for road usage. As an example, where I am, the City that I'm closest to has the rule that if you're pedalling a bicycle you're basically operating a vehicle so must follow those rules applicable to vehicle operation - however if you're pushing rather than riding your bicycle then you are considered a pedestrian.
Re safety: Kayty has summed up that quite nicely. I would add that to increase your safety on that short stretch of road in particular, you should be wearing helmet and vest if possible; you should consider the time of day you plan on heading down the road (pick times when traffic volume is down eg 2:00 in the afternoon might be a quiet time because most people are at work and 5:30 in the afternoon might be a busy time because people are coming home); consider leading him down the road (if he's quiet and steady at that) and mounting when you hit the trail; at every opportunity work on your leg yielding, turning on forehand, haunches, sidepassing, etc. to refine and improve the communication between you two so that it becomes second nature for your horse to respond to your voice, legs and reins to improve "placement" when on the road - add this to putting on lots of good miles on your horse and you've got yourself a plan.