Inferring Conceptual Knowledge from Unstructured Student Writing






Vivienne L. Ming1,2

Norma C. Ming3,4


1Socos LLC

3Nexus Research & Policy Center


Berkeley, CA 94703

San Francisco, CA 94105



2Redwood Center for

3Graduate School of Education



Theoretical Neuroscience

UC Berkeley

8UC BerkeleyBerkeley, CA 94720

9    Effective instruction depends on formative assessment to discover and monitor student

10understanding [1]. By revealing what students already know and what they need to learn, it

11enables teachers to build on existing knowledge and provide appropriate scaffolding [2]. If

12such information is both timely and specific, it can serve as valuable feedback to teachers

13and students and improve achievement [3][4]. Yet incorporating and interpreting ongoing,

14meaningful assessment into the learning environment remains a challenge for many reasons

15[5], and testing is often intrusive, demanding that teachers interrupt their regular instruction

16to administer the test. Our proposed solution to these problems is to build a system which

17relies on the wealth of unstructured data that students generate from the learning activities

18    their teachers already use. Using machine intelligence to analyze large quantities of

19passively collected data can free up instructors’ time to focus on improving their instruction,

20informed by their own data as well as those of other teachers and students. Building an

21    assessment tool which they can invisibly layer atop their chosen instructional methods

22affords them both autonomy and information. Here we present a model for inferring domain-

23specific conceptual knowledge directly from unstructured student writing from online class

24    discussion forums. Using only student discussion data from two unrelated courses –

25undergraduate introduction to biology and economics for MBA – we applied a hierarchical

26Bayesian model to infer both domain-specific concept hierarchies and individual conceptual

27patterns across students. This model extends hierarchical latent Dirichlet allocation (hLDA)

28[6] for concept modeling with a student-specific factorial model for capturing conceptual

29patterns across students. As a proof of concept, we predicted end-of-course grades, although

30the same approach may be applied to many other assessments and adaptive interventions

31recommendation. The model produced significantly better predictions compared with other

32    topic modeling techniques from the very first week of instruction, and the accuracy

33improved with additional data collected over the duration of the course (Figure Figure).

34Furthermore, additional examination of the data also reveals that higher course grades are

35correlated with a slightly higher mean of the depth parameter in hLDA. Topics in the hLDA

36model are structured in a hierarchy learned from the data, with more specialized topics being

37represented deeper in the hierarchy than more general topics. Figure Figure depicts the

38percentage of n-grams used by students receiving letter grades of A, B, and C at each of the

39four depth levels specified in the hierarchy. As shown, most of the language used by students

40    who receive C’s resides at the topmost (most generic) level, while relatively greater

41percentages of the language used by students receiving A’s and B’s reside at deeper levels in

42the hierarchy. Finally, the conceptual layer showed consistent temporal patterns of concept

43expression within each course were strong predictors of student performance and allowed


easy clustering of students for potential targeted intervention. These results suggest the


feasibility of mining student data to derive conceptual hierarchies and indicate that topic

46modeling of student-generated text may offer useful formative assessment information about

47students’ conceptual knowledge.

Figure . Accuracy of pLSA and hLDA in predicting students’ final grades from the topics in their discussion posts (MAD = mean absolute deviation).

Figure . Correlation between language depth and course grade.

50[1] J. W. Pellegrino, N. Chudowsky, and R. Glaser, Knowing What Students Know: The Science and Design

51of Educational Assessment. Washington, DC: National Academy Press, 2001.

52[2] L. A. Shepard, The role of classroom assessment in teaching and learning. (CSE Technical Report 517).

53Los Angeles CA: Center for the Study of Evaluation, 2000.

54[3] A. N. Kluger and A. DeNisi, “Effects of feedback intervention on performance,” Psychological Bulletin,

55vol. 119(2), 1996, pp. 254-284.

56[4] P. Black and D. Wiliam, Assessment and classroom learning, in Assessment in Education: Principles,

57Policy, and Practice, vol. 5(1), 1998, pp. 7-74.

58[5] M. C. Ellwein and M. E. Graue, “Assessment as a way of knowing children,” in Making schooling

59multicultural: Campus and classroom, C. A. Grant and M. L. Gomez, Eds. Englewood Cliffs, NJ:

60Merrill, 1996.

61[6] D. Blei, T. Griffiths, and M. Jordan, “The nested Chinese restaurant process and Bayesian nonparametric

62inference of topic hierarchies,” Journal of the Association for Computing Machinery, vol. 57(2), 2010,

63pp. 1-30