Well, it definitely takes consistency and time to keep horses. Almost all horses need consistent handling to keep all the natural behaviors at bay. Sounds like your horse is a bit herd bound. You don't have to ride to keep them fresh tho. I think Letagirlshowu gave you great advice.
And I would add that you never have to continue to ride if a horse is making you feel unsafe. You can get off and work on the problem on the ground. I have done that many times because I am recovering from a really bad accident ( motorcycle) and don't always feel strong enough to handle very situation. My mare was kicking out at the canter. I didnt feel confident riding her through it, which is what most folks advised me to do. So every time she kicked out, I got off quickly and lunged her at the canter with speed and intensity. She soon stopped.
When you don't have the time or energy for riding, just work on the ground with grooming, handling,leading, sending, yielding, bending, tacking up, sacking out, and games. All thi will help keep your horse tuned in to you and gentle.