Colloquium - Hinton

It is possible to combine multiple non-linear probabilistic models of the same data by multiplying the probability distributions together and then renormalizing. This is a very efficient way to model data which simultaneously satisfies many different constraints. Each individual expert model can focus on giving high probability to data vectors that satisfy just one of the constraints. Data vectors that satisfy this one constraint but violate other constraints will be ruled out by their low probability under the other expert models. For example, one expert can generate images that have the approximate overall shape of the digit 2 and other more local experts can ensure that local image patches contain segments of stroke with the correct fine structure. Or one expert model of a word string can ensure that the tenses agree and another can ensure that the number agrees.

Inference is very simple in a product of experts because the latent variables of different experts are conditionally independent given the data. However, maximum likelihood fitting of a product of experts is difficult because, in addition to maximizing the log probabilities that each expert assigns to the observed data, it is necessary to make the experts disagree as much as possible on unobserved data and so tedious Monte Carlo methods are required to compute the derivatives of the log of the normalization term. Fortunately, there is a very efficient alternative to maximum likelihood fitting which works remarkably well. Some examples of product of expert models trained in this way will be described. Products of experts work very well for handwritten digit recognition and the same algorithm can be used to fit products of Hidden Markov Models, which can have exponentially more representational power than single Hidden Markov Models.

Hosted by Michael Mozer.
Sponsored by Athene Software.