Syllabus
Probabilistic Models of
Human and Artificial Intelligence

CSCI 7222 / CSCI 4202
Fall 2010

Tu, Th 14:00-15:15
ECCR 116

Instructor

Professor Michael Mozer
Department of Computer Science
Engineering Center Office Tower 7-41
(303) 492-4103
Office Hours: Tu, Th 12:30-13:30

Course Objectives

A new paradigm has emerged in cognitive science and artificial intelligence which views the mind as a computer extraordinarily tuned to the statistics of the environment in which it operates, and views learning and adaptation in terms of changes to these statistics over time. The goal of the course is to understand the latest advances in theory in cognitive science and artificial intelligence that take a statistical and probabilistic perspective.

One virtue of probabilistic models is that they straddle the gap between cognitive science, artificial intelligence, and machine learning. The same methodology is useful for both understanding the brain and building intelligent computer systems. Indeed, for much of the research we'll discuss, the models contribute both to machine learning and to cognitive science. Whether your primary interest is in engineering applications of machine learning or in cognitive modeling, you'll see that there's a lot of itnerplay between the two fields.

The course participants are likely to be a diverse group of students, some with primarily an engineering/CS focus and others primarily interested in cognitive modeling (building computer simulation and mathematical models to explain human perception, thought, and learning).

Prerequisites

The course is open to any students who have some background in cognitive science or artificial intelligence. Undergrads need not have taken 3202 (AI 1). Some background in proababilty and statistics will be helpful, but iis not essential as long as you are a quick learner.

Optional Reference Texts

The required readings for the course will appear below in the class-by-class lecture plan. For students with a particular interest, I recommend two optional reference texts.

Chris Bishop's Pattern Recognition and Machine Learning
Bayesian Data Analysis by Gelman, Carlin, Stern, & Rubin
For statistical models, wikipedia is often a useful resource

Course Requirements

Readings

In the style of graduate seminars, your will be responsible to read the series of papers before class and be prepared to come into class to discuss the paper (asking clarification questions, working through the math in the paper, relating the paper to other readings, critiquing the paper, presenting original ideas related to the paper).

Written Commentaries

For some of the readings, I'll ask you to write a one-page commentary on the paper, The commentary consists of approximately one page of comments, questions, or critiques of the assigned reading(s) for that class. This page will be due the day of class, and can include one or more of the following:

a summary of what you think the main or most interesting ideas are behind the reading(s).
questions about the material for further discussion, either clarification questions or points of disagreement with the authors (``I don't see how such and such will work as the author claims...'').
comments on how the assigned reading relates to other course readings, or, if you feel ambitious and want to track down some related work in the field, how the assigned reading compares to this other work.
a critique of the work.
What are the flaws in the ideas presented? What are the limitations? Do the authors place their work in the appropriate theoretical perspective? Do the authors overstate their results? In what direction might the work be extended?

These commentaries are intended to promote careful thought about a paper before the session in which it is discussed. The point is not to give you more busy work, but rather to encourage you to jot down notes and questions as you read the papers. They will not be accepted after the class in which the paper is discussed.

Presentation

Students enrolled in the graduate version of the course are required to present a paper toward the end of the semester.

Implementations

Some people can look at an equation and just understand it. Other people--including myself--need to get dirty and muck with an implementation to really understand the equations and to develop intuitions about the model. I would like students in the course to implement small-scale versions of as many of the models we discuss as possible. As a tentative plan, I would like students to have four implementations over the course of the semester. I'll give guidance and recommendations as to which models to implement, but students can pick models they find particularly interesting as well.

For the students enrolled in the grad-level course, you will also be asked to go beyond a basic implmenetation and show that you can extend the model in some way: to a new problem domain, to look at some aspect of the model that the authors didn't focus on, to analyze the performance of the model, etc.

Semester Grades

Grades will be based roughly on the following: programming projects 40%, oral presentation 20%, class discussions 10%, written commentary on papers 30%. For undergraduates not required to give an oral presentation, the other components of the grade will be scaled up.

Class-By-Class Plan and Course Readings

Readings marked by (*) require commentaries.

Date	Activity	Reading	Lecture notes	Presenter
Aug 24	introductory meeting	Chater, Tenenbaum, & Yuille (2006)	lecture 1
Aug 26	basic probability, Bayes rule	Griffiths & Yuille (2006)	lecture 2
Aug 31	concept learning Bayesian Occam's razor	Tenenbaum (1999) Jefferys & Berger (1991)	lecture 3 Assignment 1
Sep 2	continuous distributions		lecture 4
Sep 7	motion illusions as optimal percepts	Weiss, Simoncelli, Adelson (2002) Weiss demo	lecture 5
Sep 9	Bayes nets I: Representation and inference	Cowell (1999) Jordan & Weiss (2002)	lecture 6 Assignment 2
Sep 14	Bayes nets II: Inference and approximate inference	optional: Huang & Darwiche, (1994)	lecture 7
Sep 16	Bayes nets III: approximate inference	Andrieu et al. (2003)	lecture 8
Sep 21	Bayes nets IV: approximate inference & learning	Andrieu et al. (2003)	Assignment 3
Sep 23	Bayes nets V: learning	Heckerman (1995)	lecture 9
Sep 28	Bayes nets VI: learning	Heckerman (1995)	lecture 10 Assignment 4
Sep 30	text mining latent Dirichlet allocation	Griffiths & Steyvers (2002) optional: Griffiths, Steyvers & Tenenbaum (2007) Blei, Ng, & Jordan (2003)	lecture 11
Oct 5	text mining applications of LDA		lecture 12	Dan Knights
Oct 7	text mining applications of LDA	recommended: video tutorial on Dirichlet Processes by Teh or Teh introductory paper		Rob Lindsey
Oct 12	text mining Inferring social networks	McCallum, Corrado-Emmanuel, & Wang (2005)	lecture 14 Assignment 5
Oct 15	nonparametric Bayes	Orbanz & Teh (2010)	lecture 15
Oct 19	text mining hierarchical models	Teh (2006)	lecture 16
Oct 21	sequential models hidden markov models	Gharamani (2001)	lecture 17
Oct 26	sequential models conditional random fields	Sutton & McCallum optional: Mozer et al. (2010) Lafferty, McCallum, Pereira (2001)	lecture 18 Assignment 6
Oct 28	sequential models Kalman filters	Koerding, Tenenbaum, & Shadmehr (2007)	lecture 19
Nov 2	sequential models particle filters, changepoint detection	Adams & MacKay (2008) optional: Wagle & Frew (2010)	lecture 20	Neeti Wagle
Nov 4	sequential models sequential dependencies	Yu & Cohen (2009)	lecture 21 Assignment 7
Nov 9	vision/attention multiple object tracking	Vul, Frank, Alvarez, & Tenenbaum (2009)	lecture 22
Nov 11	vision/attention search	Mozer & Baldwin (2008)	lecture 23
Nov 16	vision/attention search	Najemnik & Geisler, (2005) supplemental material	lecture 24
Nov 18	vision/attention focused attention	Dayan (2009)	lecture 25 Assignment 8
Nov 23	Thanksgiving break -- NO CLASS
Nov 25	Thanksgiving break -- NO CLASS
Nov 30	linguistics adaptor grammars	Johnson, Griffiths, & Goldwater optional: Johnson (2008)		William Headden, JDPA
Dec 2	linguistics coreference	Haghighi & Klein (2010) background:Haghighi & Klein (2009)	lecture 27 part 1 lecture 27 part 2	Greg Brown
Dec 7	NIPS Conference -- NO CLASS
Dec 9	NIPS Conference -- NO CLASS
Dec 14, 16:30-19:00	grad student presentations + pizza			schedule TBA

Queue

The queue is a list of papers that we haven't yet scheduled but that I hope to cover during the semester

Language

Levy

Vision

natural scene statistics (Torralba, Simoncelli, Olshausen, Zhang & Cottrell)

Other

Deep belief nets
Gaussian processes Williams (1997)
structure learning (Kemp, Tenenbaum, Griffiths, etc.)

Interesting Links

Tutorials

Owen Lewis's review of probabilistic models
Josh Tenenbaum's Bayesian models tutorial at NIPS 2006
Carl Rasmussen's Gaussian Processes tutorial at NIPS 2006
Jordan tutorial on hierarchical Dirichlet processes
Andrew Moore's tutorials
Inference in belief networks: A procedural guide (Huang & Darwiche, 1994)
Bayesian inference with tears (Kevin Knight) -- particularly useful for those interested in NLP
video lecture from summer school
Draft of nice textbook by David Barber (with matlab software)
Rob Lindsey's notes on basic probability and statistics

Modeling tools

Topic modeling toolbox (requires 32-bit matlab)
Mallet (machine learning for language, Java based implementation of topic modeling)
Mahout (Java API that does topic modeling)
C implementatoni of topic models
windows executable of C implementation (runs from the command line)
UCLA's samiam
Murphy's Bayes net toolbox
BUGS
OpenBayes
Orange
Bayesian reasoning and machine learning software in matlab (associated with David Barber's book)
Chris DeHoust comments on software

Syllabus Probabilistic Models of Human and Artificial Intelligence

CSCI 7222 / CSCI 4202 Fall 2010

Tu, Th 14:00-15:15 ECCR 116