The only time in my life that I voluntarily sold my horse was one that didn't love me. Before I bought that horse, my forever horse, whom I trained from a baby, had blossomed into an amazing show horse, winning everything in sight. At the peak of his career, he was hit by a car and killed. I was devastated. I couldn't get over it.
I bought a bunch of project horses and trained them, but that wasn't helping me. I was fighting a severe depression. Every day after work, when there was no special horse to head home to, I would put my head on my desk and sob.
So I decided to buy another forever horse. I found one that was everything I could ask for . . . so I bought him. But he walked away when I went out to the pasture to get him. He didn't like being touched or groomed or handled. Like your horse, OP, he was lovely to ride, but he could care less about me. At that time in my life, I needed a horse that would adore me, heal my broken heart, be my forever love . . . and he wasn't fulfilling that role.
So I sold him. I bought a yearling filly that I couldn't ride and named her Magic, hoping that she would work some magic in my life. She did. She adored me. If people were trying to hold her while I took care of something, she would jerk away from them, run to me and put her head on my shoulder. I kept her all her life.
If I had been in a different place in my life journey, that horse I sold would have worked out perfectly . . . but I wasn't. I needed an extra special horse, not an aloof one. I had all the hurt I could tolerate at that time. I see no harm in selling your horse and getting one who is right for you right now.