Well they can be a lot more difficult off the track than they are to break!!! Most of them, all their training is geared towards going really fast, and although there is some directional control and some speed control (speed control is more in the stayers than the sprinters) many of them think that canter does not exist.
On the other hand with a fresh breaker, it's your job to teach them that walk/trot/canter are ok, and you're not going to eat them when you sit on their back. Getting them used to the tack can be tricky (my filly is ear shy so bridling is difficult on a good day, impossible on a bad day), mouthing them usually isn't difficult as long as you have the feel and timing - but mouthing is something that most ex racehorses need so I bet you've done that before. There are a lot of things that a breaker doesn't understand that a racehorse does.
It's a different set of challenges but once the breaking itself is through I don't think you'll struggle. Ex racehorses are far more challenging than something greenbroke that hasn't been raced, purely because you have all that racetrack training that you have to undo. Different training for different uses!
A CLEAN SLATE FOR THE FUTURE