This full-day short course introduces a selection of statistical learning methods for analyzing process data, that is, log data from computer-based assessments. Covered topics include (1) data-driven methods for extracting features from response processes via n-grams, multidimensional scaling, and sequence-to-sequence autoencoders; (2) introduction to ProcData, an R package for process data analysis; and (3) applications of process features to practical testing and learning problems, including scoring, differential item functioning correction, and computerized adaptive testing. Mode of instruction will be a blend of presentations, for topic (1), and concrete illustrations in R, for topics (2) and (3). Intended audience are researchers and practitioners interested in data-driven methods for analyzing process data from assessments and learning environments. To fully engage in the hands-on activities, familiarity with R and RStudio is expected. Running the ProcData package requires installation of R and Python. Installation instructions and support will be provided. Participants are expected to bring their own laptop with Windows or Mac operating system. By the end of the course, participants are expected to get a composite picture of process data analysis and know how to conduct various analyses using the ProcData package.
Registration is required through 2021 IMPS website.
This short course will be held on July 19, 2021. Below is the course schedule. All times are in Eastern Daylight Time (EDT).
| Time | Lecturer | Session Title |
|---|---|---|
| 10:00 am — 10:15 am | Qiwei He | Opening remarks |
| 10:15 am — 11:45 am | Jingchen Liu | Overview of process data analysis |
| 12:00 pm — 01:30 pm | Qiwei He | Introduction to n-grams and longest common subsequence |
| 01:45 pm — 03:15 pm | Xueying Tang | Introduction to ProcData package |
| 03:30 pm — 05:00 pm | Susu Zhang | Partial scoring and DIF correction |
Recent advances in informational technology has led to the increasing popularity of computer-based interactive items, which require test-takers to complete specific tasks within a simulated environment. In addition to the final outcomes, the entire log of interactions between the test-taker and the item, i.e., the sequence of actions and their timestamps, are recorded as process data. Process data contain rich information about test-takers’ problem-solving processes that are not recoverable from the final responses. In this overview, we summarize our main research developments of process data analysis. It includes feature extraction via multidimensional scaling and neural-network-based autoencoder. An important question is how process data can assist specific psychometric research. To address this problem, we present two applications: improving test reliability by constructing a process-data-based partial score system and removing/reducing differential item functioning by including process data in the scoring rules.
We introduce two approaches to handle the sequential process data to derive informative features. The topics covered in this session include
We introduce ProcData, an R package we design for processing, examining, and analyzing process data. The topics covered in this session include
proc and its print and summary methodscc_dataread.seqs and write.seqsseq2feature_mdsseq2feature_seq2seqseqmWe will demonstrate the features of ProcData through live R sessions.
We provide two specific applications of process data analysis to psychometric problems. These two examples illustrate how to make use of the additional information in process data and to what extent they add values to the existing literature.
Accurate assessment of examinees’ abilities is the key task of testing. Traditional assessments are based on the item final responses, while problem-solving processes contain additional information about a student’s proficiency on the measured trait. We establish a framework to systematically construct a process-data-based scoring system that is substantially more accurate than the traditional IRT-model-based assessment in terms of reliability.
Differential item functioning (DIF) can jeopardize test fairness and validity. Various methods have been developed to identify DIF. However, few results are available to reduce or correct DIF. We develop a framework that identifies and further constructs a scoring rule to reduce DIF. This new scoring rule is based on an individualized score adjustment with process data.
In this section, we provide a step-by-step instruction of these two methods through simulated data.
Dr. Qiwei (Britt) He is Research Scientist in the Center for Psychometrics and Data Science Modeling at Educational Testing Service (ETS). She holds a PhD of Psychometrics and Data Science from University of Twente, Netherlands. She has research interests in educational and psychological measurement, data/text mining, with specific attention to complex new data source in computer-based tests (e.g., process data, textual data) and methodology advancement in large scale assessments (e.g., PISA, PIAAC). She is the recipient of 2019 Jason Millman Promising Measurement Scholar Award given by the National Council on Measurement in Education (NCME), 2017 Alicia Cascallar NCME Award for an Outstanding Paper by an Early Career Scholar and the OECD Thomas J. Alexander Fellowship. She is leading an OECD project in leveraging process data in dynamic navigation of reading from multiple-source texts and co-leading an NSF-funded project to develop latent and graphical models for complex dependent data in education. Email: qhe@ets.org
Dr. Jingchen Liu is Professor of Statistics at Columbia University. He holds a Ph.D. in Statistics from Harvard University. He is the recipient of 2018 Early Career Award given by the Psychometric Society, 2013 Tweedie New Researcher Award given by the Institute of Mathematical Statistics, and a recipient of the 2009 Best Publication in Applied Probability Award given by the INFORMS Applied Probability Society. He has research interests in statistics, psychometrics, applied probability, and Monte Carlo methods. He is currently an associate editor of Psychometrika, British Journal of Mathematical and Statistical Psychology, Journal of Applied Probability/Advances in Applied Probability, Extremes, Operations Research Letters, and STAT. Email: jcliu@stat.columbia.edu
Dr. Xueying Tang is an Assistant Professor in Statistics in the Department of Mathematics at the University of Arizona. Prior to joining the University of Arizona, she was a postdoctoral research scientist at Columbia University in the Department of Statistics. Her research interests include high dimensional Bayesian statistics, latent variable models and their applications in education and psychology. Email: xytang@math.arizona.edu
Dr. Susu Zhang is an Assistant Professor of Psychology and Statistics at the University of Illinois at Urbana-Champaign. Her research interests include latent variable modeling, the analysis of complex data (e.g., log data) in computer-based educational and psychological assessments, and longitudinal models for learning and interventions. Email: szhan105@illinois.edu