As another returning adult rider, I can tell you that trying to do both hunt seat AND dressage is torture. If you really, really want to make the switch to dressage, you might have to commit to it 100%. I absolutely could not do both. Doing hunt seat reinforced my old habits, and undid any new dressage techniques that I was trying to learn.
This is my experience as well. Except for me, it was my hunter equitation coach pulling his hair out over my corrupted forward seat.
While it's definitely hard, it's not impossible. Especially if you don't intend to show seriously. There's definite benefits to training in other disciplines. But you have to be aware that your progress in the dressage ring (and possibly hunter ring) might be slowed. Just make sure your instructors know what your doing, and ask if they have any advice on making the transition between the two less difficult.
Some random thoughts:
-- I'm 32 and I just got back into riding after a 10-year hiatus and things seem to upset me much more now than they did back when I first started riding when I was 19. If the horse is slow or bouncy, if the horse spooks ... if my trainer says something and I take it personally. My trainer and I get along very, very well, but even I have days when I think she hates me. And the horse hates me. And the weather hates me ... And the next lesson, I'll be fine. Maybe it's hormones, maybe it's stress from work, or whatever. But it passes, usually by the next lesson. I'm actually making an effort now to not dwell on bad lessons, unless they continue for more than a couple of weeks. Thankfully, it never has.
-- About your goals: I write down my goals every 6 months. If they've changed, I share them with my instructor. In fact, after I sent her my goals in June, she made all her lesson students do the same. I think it's important for students/riders to know what they want to accomplish so they're not floundering around. And I do it every six months because my goals can change and I not even be aware of it unless I think about it. Plus, it could help your instructors cater your lessons better. For me, I want to show eventually, just not right now. As such, my trainer throws in some dressage lessons and wants to take me out into the cross country field soon for my lessons which suits me just fine.