The search for a common ancestor linking modern humans with the Neanderthals who lived in Europe thousands of years ago has been a compelling subject for research. But a new study suggests the quest isn't nearly complete.

The researchers, using quantitative methods focused on the shape of dental fossils, find that none of the usual suspects fits the expected profile of an ancestor of Neanderthals and modern humans. They also present evidence that the lines that led to Neanderthals and modern humans diverged nearly 1 million years ago, much earlier than studies based on molecular evidence have suggested.

The study, which will be published online this week by the Proceedings of the National Academy of Sciences, was carried out by an international team of scholars from The George Washington University, Konrad Lorenz Institute for Evolution and Cognition Research in Austria, Indiana University and Atapuerca Research Team in Spain.

"Our results call attention to the strong discrepancies between molecular and paleontological estimates of the divergence time between Neanderthals and modern humans," said Aida Gomez-Robles, lead author of the paper and a postdoctoral scientist at the Center for the Advanced Study of Hominid Paleobiology of The George Washington University. "These discrepancies cannot be simply ignored, but they have to be somehow reconciled."

P. David Polly, professor in the Department of Geological Sciences in the IU Bloomington College of Arts and Sciences, is a co-author of the study. Other co-authors are Spanish researchers Jose Maria Bermudez de Castro, Juan-Luis Arsuaga and Eudald Carbonell, co-directors of the excavations at Atapuerca sites.

The study resulted from a collaboration that developed when Gomez-Robles spent a semester at IU studying with Polly while she was a graduate student at the National Research Centre for Human Evolution and at the University of Granada, both in Spain. It also makes use of statistical methods developed by IU Bloomington biologist Emilia Martins.

The article, "No known hominin species matches the expected dental morphology of the last common ancestor of Neanderthals and modern humans," relies on fossils of approximately 1,200 molars and premolars from 13 species or types of hominins -- humans and human relatives and ancestors. Fossils from the well-known Atapuerca sites have a crucial role in this research, accounting for more than 15 percent of the complete studied fossil collection.

The researchers use techniques of morphometric analysis and phylogenetic statistics to reconstruct the dental morphology of the last common ancestor of Neanderthals and modern humans. They conclude with high statistical confidence that none of the hominins usually proposed as a common ancestor, such as Homo heidelbergensis, H. erectus and H. antecessor, is a satisfactory match.

"None of the species that have been previously suggested as the last common ancestor of Neanderthals and modern humans has a dental morphology that is fully compatible with the expected morphology of this ancestor," Gomez-Robles said.

The study also finds that the potential human ancestors discovered in Europe are morphologically closer to Neanderthals than to modern humans. This suggests the line leading to Neanderthals arose around 1 million years ago and the divergence of humans took place much earlier than previously thought. Other studies have placed the divergence around 350,000 years ago.

The researchers argue that quantitative and statistical methods provide a better way to settle debates about human origins than the descriptive analyses that have been used in the past. "Our primary aim," they write, "is to put questions about human evolution into a testable, quantitative framework and to offer an objective means to sort out apparently unsolvable debates about hominin phylogeny." They also suggest applying their methodology to study other body parts represented in the hominin fossil record.

What comes next? Answers to the ancestry question could come from studying hominin fossils from Africa, the researchers say. But the African fossil record from the era of interest is sparse.

"The study tells us that there are still new hominin finds waiting to be made," Polly said. "Fossil finds from about 1 million years ago in Africa deserve close scrutiny as the possible ancestor of Neanderthals and modern humans."