skip to main content
Department of Computer Science University of Colorado Boulder
cu: home | engineering | mycuinfo | about | cu a-z | search cu | contact cu cs: about | calendar | directory | catalog | schedules | mobile | contact cs
home · events · colloquia · 2000-2001 · 

Colloquium - Hinton

ECCR 200

Products of Experts
University College London - Gatsby Computational Neuroscience Unit

It is possible to combine multiple non-linear probabilistic models of the same data by multiplying the probability distributions together and then renormalizing. This is a very efficient way to model data which simultaneously satisfies many different constraints. Each individual expert model can focus on giving high probability to data vectors that satisfy just one of the constraints. Data vectors that satisfy this one constraint but violate other constraints will be ruled out by their low probability under the other expert models. For example, one expert can generate images that have the approximate overall shape of the digit 2 and other more local experts can ensure that local image patches contain segments of stroke with the correct fine structure. Or one expert model of a word string can ensure that the tenses agree and another can ensure that the number agrees.

Inference is very simple in a product of experts because the latent variables of different experts are conditionally independent given the data. However, maximum likelihood fitting of a product of experts is difficult because, in addition to maximizing the log probabilities that each expert assigns to the observed data, it is necessary to make the experts disagree as much as possible on unobserved data and so tedious Monte Carlo methods are required to compute the derivatives of the log of the normalization term. Fortunately, there is a very efficient alternative to maximum likelihood fitting which works remarkably well. Some examples of product of expert models trained in this way will be described. Products of experts work very well for handwritten digit recognition and the same algorithm can be used to fit products of Hidden Markov Models, which can have exponentially more representational power than single Hidden Markov Models.

Hosted by Michael Mozer.
Sponsored by Athene Software.

The Department holds colloquia throughout the Fall and Spring semesters. These colloquia, open to the public, are typically held on Thursday afternoons, but sometimes occur at other times as well. If you would like to receive email notification of upcoming colloquia, subscribe to our Colloquia Mailing List. If you would like to schedule a colloquium, see Colloquium Scheduling.

Sign language interpreters are available upon request. Please contact Stephanie Morris at least five days prior to the colloquium.

See also:
Department of Computer Science
College of Engineering and Applied Science
University of Colorado Boulder
Boulder, CO 80309-0430 USA
Send email to

Engineering Center Office Tower
ECOT 717
FAX +1-303-492-2844
XHTML 1.0/CSS2 ©2012 Regents of the University of Colorado
Privacy · Legal · Trademarks
May 5, 2012 (13:29)