I don't think its really scarier than riding, it just depends on what you are used to. I grew up with teams and riding horses both. I haven't particularly liked riding in the wagon for several years simply because I feel out of control if I can't actually get my hands on the horse. I have been hurt much worse by saddle horses than I ever was by a team but that feeling of disconnection between me and the team kinda freaks me out. Now, whenever Dad takes the team for a drive, I stick with my saddle horse beside the wagon.