I always ride alone. I find I feel more in tune with my horse, and I enjoy our time more. I just can't get coordinated to meet with friends, I dn't have a trailer to travel to some other location.
Start off with very short rides that you will be comfortable with. Take a cell phone, let someone know where you riding even on short rides. Even if my husband is not home, I leave a note with where I'm going and what time I left, just in case I don't come back
Any time you truly are scared, don't go out. Try again some other day. Mix it up, ride with a friend one day and alone the next. And it doesn't have to be hours-long. I ride before work so most of my rides are less than 1/2 hour. It's getting out that's important, not the duration.