Syllabus
Probabilistic Models of
Human and Machine Intelligence
CSCI
7222
Fall 2015
Instructor
Professor
Michael
Mozer
Department of Computer Science
Engineering Center Office Tower 741
(303) 4924103
Office Hours: Th 13:0014:30
Course Objectives
A
dominant
paradigm in artificial intelligence and cognitive science views the
mind as a computer extraordinarily well tuned to statistics of
the
environment in which it operates. From this perspective, learningby
both humans and machinesinvolves collecting observations and updating
statistics. The
goal of the course is
to understand
the latest advances in theory in artificial intelligence and cognitive
science that take a statistical and probabilistic perspective on
learning and intelligence.
One virtue of probabilistic models is that they
straddle
the gap between cognitive science, artificial intelligence, and machine
learning. The same
methodology
is useful for both understanding
the brain and building intelligent
computer systems. Indeed, for much of the research we'll
discuss,
the models contribute both to machine learning and to cognitive
science. Whether your primary interest is in engineering
applications of machine learning or in cognitive modeling, you'll see
that there's a lot of interplay between the two fields.
The course participants are likely to
be a
diverse group of students, some with primarily an
engineering/CS focus and others primarily interested in cognitive
modeling (building computer simulation and mathematical models to
explain human perception, thought, and learning).
Prerequisites
The
course is open to any
students who
have some background in cognitive science or artificial intelligence
and who have taken an introductory probability/statistics course.
If your background in probability/statistics is weak, you'll
have
to do some catching up with the text.
Course Readings
We
will be using a text by David Barber (
Bayesian
Reasoning And Machine Learning,
Cambridge University Press, 2012.
The author has made available an
electronic
version of the text. Note that
the electronic version is a 2015
revision. Because the electronic version is more recent, all reading
assignments will refer to section numbers in the electronic version.
For additional references,
wikipedia
is
often a useful resource. The pages on various probability
distributions are great references. If you want additional reading, I
recommend the following texts:
We will also be reading research articles from the literature, which
can be downloaded from the links on the classbyclass syllabus below.
Course Discussions
We
will use Piazza for class discussion.
Rather than emailing me, I encourage you to post your
questions
on Piazza. I strive to respond quickly. If I do not, please
email
me personally. To sign up, go
here.
The class
home page is
here.
Course
Requirements
Readings
In
the style
of graduate
seminars, your will be responsible to read
chapters from the text and research articles before
class and be prepared to come into class to discuss the material
(asking
clarification questions, working through the math,
relating papers to each other, critiquing the
papers, presenting
original ideas related to the paper).
Homework Assignments
We can
all delude ourselves into believing we
understand some math or algorithm by reading, but implementing and
experimenting with the algorithm is both fun and valuable for obtaining
a true understanding. Students will implement smallscale
versions of as many of the models
we discuss as possible. I will give about 10 homework
assignments
that involve implementation over the semester, details to be
determined. My preference is for you to work in matlab, both because
you can leverage software available with the Barber text, and because
matlab has become the de facto work horse in machine learning.
For one or two assignments, I'll ask you to write a onepage
commentary on a research article.
Semester
Grades
Semester
grades will be based 5% on class
attendance and participation and 95% on the homework assignments.
I will weight the assignments
in
proportion to their difficulty, in the range of 5% to 10% of the course
grade. Students with backgrounds in the area and specific
expertise may wish to do inclass presentations for extra credit.
ClassByClass Plan and Course
Readings
The
greyed out portion of this schedule is tentative and will be adjusted
as the semester goes on. I may adjust assignments, assignment dates,
and lecture topics based on the class's interests. Due dates for an
assignment will be the date that the next assignment is handed out.
Date 
Activity 
Required
Reading
(Section numbers refer to 2015 edition of Barber) 
Optional
Reading 
Lecture
Notes 
Assignments

Aug
25 
introductory
meeting 
Appendix
A.1A.4,
13.113.4 
Chater,
Tenenbaum, & Yuille (2006) 
lecture 

Aug 27 
basic
probability, Bayes rule 
1.11.5,
10.1 
Griffiths
&
Yuille (2006) 
lecture 
Assignment 0 
Sep 1 
continuous
distributions

8.18.3 

lecture


Sep 3 
concept
learning,
Bayesian Occam's razor 
12.112.3
(omit 12.2.2,
which requires some probability we haven't yet talked about) 
Tenenbaum
(1999)
Jefferys
& Berger (1991) 
lecture 
Assignment
1 
Sep 8 
Gaussians

8.48.5 

lecture 

Sep 10 
motion
illusions as optimal
percepts

Weiss,
Simoncelli,
Adelson (2002) 
motion
demo 1
motion
demo 2

lecture 
Assignment 2 
Sep
15 
Bayesian
statistics
(conjugate priors, hierarchical Bayes) 
9.1 
useful reference:
Murphy (2007) 
lecture 

Sep 17 
Bayes
nets: Representation 
2.12.3,
3.13.5

Cowell (1999)
Jordan
& Weiss (2002)
4.14.6

lecture

Assignment 3

Sep 22 
Bayes
nets: Exact Inference

5.15.5 
Huang
& Darwiche (1994)

lecture


Sep 24 
Assignment 4 
Sep 29 
Bayes
nets: Approximate
inference 
27.127.6 
Andrieu
et al.
(2003) 
lecture


Oct 1 

Oct 6 
<catch up
day> 



Assignment 5 
Oct 8 
Learning
I: Parameter learning

8.6, 9.29.4 
Heckerman
(1995)
9.5 
lecture 

Oct 13 
Learning II:
Missing data, latent variables, EM, GMM 
11.14, 20.13 

lecture 

Oct 15 
text mining
latent
Dirichlet
allocation 
20.6 
Griffiths,
Steyvers
& Tenenbaum (2007)
Blei,
Ng, & Jordan (2003)
video
tutorial
on Dirichlet Processes by Teh or Teh
introductory paper 
lecture 
Assignment 6 
Oct 20 
text mining
variational
methods
GUEST LECTURER: Jordan BoydGraber

28.128.5, 11.5 
28.628.9 
lecture1
lecture2 

Oct 22 
Oct 27 
text mining
topic model
extensions 
McCallum,
CorradoEmmanuel, & Wang (2005) 
Bamman,
Underwood, & Smith (2014)

lecture


Oct 29 
text mining
nonparametric
Bayes
hierarchical models 
Orbanz &
Teh (2010)
Teh
(2006) 

lecture1
lecture2 

Nov 3 
Assignment 7 
Nov 5 
modeling and
optimization
Gaussian processes 
19.119.5 

lecture1
lecture2 

Nov 10 
modeling and
optimization
Multiarm bandits and Bayesian optimization 

Shahriari, Swersky, Wang, Adams, and
de Freitas 
lecture 

Nov 12 
modeling and
optimization\
Guest
speakers: Mohammad Khajah, Manjhunath Ravi 



Assignment 8 
Nov 17 
sequential models
hidden Markov models
conditional random fields 
23.123.5 
Gharamani
(2001)
Sutton &
McCallum
Mozer
et
al. (2010)
Lafferty,
McCallum, Pereira (2001) 
lecture 1
lecture 2 

Nov 19 
sequential models
exact and approximate
inference (particle filters,
changepoint detection) 
27.6
Adams
& MacKay (2008)
Yu
& Cohen (2009) 
Wilder,
Jones, & Mozer (2010)

ppt
pdf 

Dec 1 
sequential models
Kalman filters 
24.124.4 
Koerding,
Tenenbaum, & Shadmehr (2007)
24.5 
lecture 

Dec 3 
vision/attention
search 
Mozer
& Baldwin (2008)
Najemnik
& Geisler (2005) 
supplemental
material for Najemnik & Geisler 
lecture
lecture 
Assignment 9 
Dec 8 
CLASS
CANCELLED
(or
guest lecture) 




Dec 10 
CLASS CANCELLED (or guest
lecture) 




Dec 14
13:3016:00 
Final
project
presentations 



Assignment 9 due 
Queue
Interesting
Links