Hello and welcome to the forum!
First of all, do you have a video you can show us?
Secondly, I'd advise you to check out the "sticky" on the forum that's got great drills and exercises for barrel racers of all levels. Barrel Racing Exercises and Drills.
Now, onto your "problem". How exactly are you "holding him off" the barrel? I am going to make the assumption that you are making the very common mistake of trying to turn his head away from the barrel, to keep him from turning too soon. THAT is why he is shouldering into the barrel.
When you approach the barrel, a good rule of thumb is that you should head for a spot (some call it an axis point) that is 4 feet away from the barrel. From when you start the turn, until you finish, your horse's body needs to have bend in it. So as you approach the barrel, you should "lift" with your inside rein (making a motion to twist your wrist, and move it in the direction of your opposite shoulder) and ask your horse to tip their nose slightly inward. Use your outside leg, slightly back, to ask your horse to bring their hindquarter in and underneath themselves. This should cause your horse to wrap around your inside leg.
It should look like this:
Use your LEGS to control your horse during the turn. So if your horse wants to turn too soon, give them inside leg pressure to "hold" them. Still keep their nose tipped in, and still keep their hip in. But use your inside leg to laterally move them away from the barrel.
NEVER turn the horse's head away from the barrel to avoid turning too soon. This takes away the bend the horse needs to complete the turn. Therefore your horse is going to either 1) shoulder in and hit the barrel 2) run past the barrel and not turn at all.
Spirals are a great way to work on leg control with your horse. Use just one barrel and start loping a huge circle around it. Using only your LEGS, ask your horse to start to slowly decrease the circle size, and then slowly increase the circle size. Your horse will need to listen to you on which way to go. And make sure to keep his body properly bent the whole time.
You can also do reverse arc exercise to keep your horse anticipating. When you reach your pocket on the barrel, you are actually going to turn your horse AWAY from the barrel, but still keep the same bend in his body. So if you were going to make a right barrel turn, you are going to circle to the left, while keeping the horse's body bent around your right (inside) leg. Then when you come to the barrel again, after doing the reverse arc, your horse's body is in the proper position to turn the barrel. Again, use your LEGS; not the reins.