How careful are you when you remove the bridle? You may think you are being careful and gentle but you may be banging the horse's gums/roof of mouth/teeth as you remove the bit. Therefore he may have developed a habit of tensing up when it is removed, pulling back to get rid of it quickly or holding on to it. We've had horses with this habit before. Some horses you can do whatever with and they don't care, but some are just really sensitive to stuff like this and so develop these problems.
It might be the bit is already causing soreness (the redness you noticed) and the problem has arisen in this way because removing the bit has aggravated a problem already there.
I'm surprised no one's mentioned this explanation yet. I'm not saying it's the right one for this situation, but it's the most common reason for horses developing this problem. If it is the case it will just take a long time and lots of patience and very careful handling to get the horse used to it again.