Yes, you can. But should you? Depends on the height difference and your placement behind the horse. Minimal changes and if you are tall in the trunk it may work. Really, though the tire diameter needs to change and seat may need raising.
americandrivingsociety.org has or used to have a table that listed all relevant measures by the height of the horse. It is a good start. You still want to know how long your horse is as some short backed or longer backed could mean slight changes in certain measures.
Some horse people change their horse, they change their tack and discipline, they change their instructor; they never change themselves.