Since physicist Edwin T. Jaynes published his Maximum Entropy Principle for quantitatively modeling real-world systems with incomplete information and uncertainties, the approach proved useful in a wide variety of fields, with citations to his work doubling every ten years and showing no signs of slowing down. Jaynes, generalizing from the field of statistical mechanics, provided a principled way to draw statistical inferences and make predictions in systems from biology, sociology, chemistry, drug discovery, economics, finance, ecology, computer science, engineering and other disciplines. However, the approach sometimes fails spectacularly (called the degeneracy problem), making predictions far removed from observed reality as David Strauss demonstrated for the case of social networks in 1986.
In a breakthrough approach, solving this problem that was open for nearly thirty years, a team led by physics professor Zoltán Toroczkai has provided an understanding of the degeneracy problem and discovered a way to eliminate it. Their paper, “Reducing Degeneracy in Maximum Entropy Models of Networks,” was published this month in Physical Review Letters. Coauthors with Toroczkai are Szabolcs Horvát (first author) and Éva Czabarka.
Building on the observation by physicist Mark E. J. Newman in 2004 that degeneracy formally resembled phase transitions such as those appearing in ferromagnets or superconducting materials, the team mathematically proved that the Maximum Entropy approach fails when there are strong nonlinear correlations between different pieces of the input data used for generating inferences by the method. Exploiting this observation they then proposed certain nonlinear transformations on the data in ways that eliminated degeneracy and made the method applicable to these cases as well.
The power of the approach, like Jaynes’s, is that it is mathematically principled, unlike other ways of prediction that perform heuristic inferences extrapolated from patterns in the data. The work, sponsored by the Defense Advanced Research Projects Agency, focused on predicting behavior in social networks, but it is applicable to the same breadth of fields as the Maximum Entropy Principle itself.
“We are quite excited about it because of this full circle of notions–the method, originating from physics and information theory was applied in virtually all fields of science; now its range of applicability has been expanded even more based an understanding that again, goes back to physics,” Toroczkai says.