If you take a break it is fine but I would try to keep it up at some level - eg if you are riding daily then cut it back to weekly, if weekly cut it back to monthly - so that you don't lose touch until you are really sure you are sick of it.
And yes, the girl possibilities are huge. Here in the UK (though we love our camp dressage divas) it is hard to meet a guy who is into horses who is straight and also not a tiny, anorexic jockey. The rare straight ones who are of normal build are snapped up by very talanted, fit, expert young horsewomen. It may take a couple of years for this to pay off (horsey girls can be quite shy) but it's well worth sticking around to see how it pans out . .