Drafts and smaller horses can live well together, if introduced properly before being just dumped in together.
Drafts are absolutely hard on fences. If they lean over to get to interesting grass, their weight will push a fence down. If they rub their bums, same thing. Fence will go down. My daughter's one mare has bent more steel fence panels than I can count. Just bent them because they wouldn't go down.
Gypsies love to jump and are good at it, so will quite easily scale most fencing if they feel like it. I don't know that other draft breeds are so inclinded to jump.
Very good fencing is a must. No barbed wire of course and electric if possible.