I think it's a compliment, you have a passion for horses and it shows!
I've been told this since I started horse riding...which was like over 10 years ago! (Yay for horse riding!).
The horses I work with are my "children", my boss signed my christmas card "From.......& the ponies" :3
When people say that I just say, I know. (And then start talking about why I love horses!)