What type of paper and pastels are you using? They make a huge difference in the quality of your portrait. Particularly when starting out, you really need to use quite a heavily toothed paper in order for it to hold all of the layers on.
What colour is your horse? Say a chestnut, there are multitudes of base colours in that coat. Some have a blue base, some green, some purple, very deep red etc. I layer at least one, if not two bases of that colour down in all of my drawings. Then from there, start working towards the coat colour. So in a red based chestnut, I'd put down a light blue/purple base, lay down my shadows and highlights, and start to work my deep reds out from there. Only the final two layers are actually the true colour of the horse. If you just slap on some brown and orange, there is going to be no depth.
Also, make sure you make big use of your shadows and highlights. Make highlights white, and shadows black with a deep blue/purple base to give the black a really deep feel about it. If you start a portrait by marking out those shadows and highlights, then working around them, your portrait will take shape much more readily and it will force your eye to think more about getting depth in the colour.
I hope this helps you somewhat!