As a general answer, yes, Mg will help reduce anxiety, when, as others have said, it is deficient/imbalanced. Mg is very commonly deficient in the diet, esp give below considerations of rich feed & stressy lifestyles. As a... long winded answer...
As others have said, there are many forms of Mg & some will do a lot, some, due to not being 'bioavailable', will do little. Mg melatate is one bioavailable source, while Mg Ox(what's in most cheaper feeds & supps) is not very.
Mg is one of those nutrients which depends on many factors as to how much a body needs/uses. So there is no 'x amount for a balanced diet'. Eg. if a horse is on rich, high sugar diet, it will need more Mg. If the horse is IR/EMS/'founder prone'/obese, it will need more Mg If it is under stress, it will need more Mg. If it is physically hurting or ill, it will need more Mg. And if the diet is high Ca, it will need a lot of Mg.
And as Gotta said, it is a mineral which doesn't just effect emotions/mental health, but physically, muscles, nerves, tendons, inflammation and energy(chronic fatigue in people is linked with low Mg) so a horse may well be stressy just due to mental reasons, but they may be a bit sore, from stiff muscles or such, which may also be relieved by the Mg. And it has been associated with improved hoof health - particularly caudal foot development.
So, given last factor above, I started my horses on it years ago, to see what it did for their hooves(& they were long term fat & I knew it was good for 'resensitising' the body to insulin). I'd never tried it for behaviour, as I believed my horses had a pretty well balanced diet anyway, but one of them had... nervous issues that while I knew was also about training/experience, was somewhat inexplicable - unexpected. Within only a week or so of putting them on it, my horse had just... chilled. Gone back to his previously unflappable self, without me doing anything different training or management-wise. Then I started really looking into Mg!
In the last 8 years or so, I've had my horses on it as a general supp, and also many of my clients, friends and farriery associates have had their horses on it, and while experience is always varied, based on many factors, they have generally found a positive difference in their horses, both physically & behaviourally.