The laws regarding Common Law Marriage vary from jurisdiction to jurisdiction. Also, it IS legally binding in some jurisdictions, whereas it is not in others.
Currently there are only 11 states that recognize Common Law Marriages within the U.S. In addition, also there are other states that have grandfathered in the recognition of Common Law Marriages (established before a certain time period).
Common Law Marriages usually are not licensed by a government official, however some states require it to be 'Common Law". Cohabitation now is not enough to be 'common law', you have to proclaim yourself as the spouse to the other. In conjunction to this, both parties must consent to 'being married' and are of legal age to enter into a contract such as marriage. Also, in many states, those who recognize Common Law Marriage, require a minimal amount of time of cohabitation before a couple can be valid as common law.
On a side note: There is no such thing as Common Law Divorce.
My opinion is this, if you feel enough in 'love' to live together, but not get married and make it legal, then so be it. I am not one to judge. However, I, like many other people were raised with the theory that when you love someone, and want to spend the rest of your lives together, you get married. I don't see Common Law Marriage as wrong, but obviously some states' government does.