Yes you get penalities if your horse has a run out or refusal on cross country, it's 20.00 for the first offense at a fence, 40.00 for the second at the same fence, third at the same fence is elimination. You can have up to 4 refusals on course at different efforts before elimination, however usually people retire before that point.
You can get penalties at the lower levels for running too fast, and at all levels you get them for running too slow. You cannot continue the competition if you are eliminated in one of the phases, however SOME organizers will allow you to have a schooling round if you have a lot of trouble and you ask them. Not all will let you though.
I think you should take a look at USEA - United States Eventing Association (Eventing, Combined Training, Three Day Event, Horse Triathlon, Equestrian Sport) and do some research on the sport before getting into this.