Time management is a HUGE life lesson to master when you go off to college, whether or not you opt to include horses. If you're living away from home in a dorm or apartment, this could be the first time you have had to figure out when/where/how you will eat, sleep, study, ride, play, do laundry, and pay for it all. It's VERY easy to get off track, but once you find your groove it's all good.
I graduated from a 4 year college with both an Equine degree and a regular Business degree (good cautionary choice, as I no longer make my living working with horses!) I did not bring my own horses with me, preferring to gain more experience riding a variety of different horses at the school and through the traveling Equestrian Team. The Equestrian students have VERY full plates - in addition to juggling all the regular college classes & homework, sometimes a horse final the next morning would require a late night of bathing, clipping & braiding, or maybe you need to give that 2 year old one more quick ride. You get home exhausted, filthy, and long after the school cafeteria closed for dinner, and then you still have to sit down and study for a calculus exam.
Some of us also had to fit in part-time jobs. While I had enough grants & parental assistance to cover most of my tuition & school meal plan, I still had to cover my regular clothes, work clothes & horse showing clothes, daily toiletries, class books, notebooks & pens, tack & grooming supplies for my assigned school horses, laundry detergent & quarters for the laundromat, and the gasoline to get me where I needed to be.
That said, if horses are your passion, you will figure out how and where to fit them into your life - whether we're talking during college or after. Many people have done it and will continue to do so. There is nothing wrong with trying something for a semester and learning whether or not it works for YOU.