08/03/2026
By Sherin Bojappa Muckatira

The Kennedy College of Sciences, Miner School of Computer & Information Sciences, invites you to attend a doctoral dissertation proposal defense in computer science by Sherin Muckatira. The proposal is titled “Data-Centric Understanding of Abilities in Transformer Language Models.”

Event Details

  • Date: Aug. 24, 2026
  • Time: 10 – 11 a.m. Eastern time
  • Location: Virtual via Zoom meeting

Committee Members

  • Anna Rumshisky, advisor: Associate Professor, Miner School of Computer & Information Sciences
  • Tingjian Ge: Professor, Miner School of Computer & Information Sciences
  • Ming Shao: Associate Professor, Miner School of Computer & Information Sciences
  • Boqing Gong: Assistant Professor, Boston University

Abstract

Large language models (LLMs) are trained on vast amounts of data, with recent models trained on trillions of tokens. Their performance on benchmarks testing capabilities such as in-context learning and chain-of-thought reasoning improves with both model size and data size. Many frontier models are trained on far more tokens than predicted by the Chinchilla scaling laws, suggesting that pre-training data remains central to the emergence of model capabilities. However, the role of data is difficult to study directly in frontier-scale systems, where pre-training corpora are enormous and model training is computationally expensive.

This thesis takes a data-centric approach to understanding abilities in transformer language models. The first part of the thesis focuses on reduced-scale language models, ranging from 1M to 165M parameters, and asks whether capabilities associated with larger models can be observed in smaller models through simplification of the pre-training data. We show that downscaling the data through vocabulary-based filtering of pre-training corpora can simplify the modeled language and enable smaller generative language models to exhibit stronger zero-shot behavior on simplified downstream tasks. These results suggest that some abilities often associated with model scale can be studied in smaller models when the data distribution is correspondingly simplified.

The thesis then studies grokking-like generalization dynamics during language-model pre-training using a data-exposure-based framework. We propose a pre-training analogue of grokking by constructing exposure-based proxy splits over grammatical minimal-pair evaluations. This setup distinguishes examples whose critical phrases appear verbatim in the pre-training corpus from examples whose critical phrases are not directly observed during pre-training. By tracking model behavior across pre-training checkpoints, we show that accuracy first improves on the exposed split before improving on the unexposed split, revealing delayed generalization dynamics similar to grokking. We further analyze this transition using grammar-specific concept-vector representations, providing insight into how internal representations evolve during pre-training.

The second part of the thesis turns to evaluation data. Many existing mathematical benchmarks evaluate language models using final-answer correctness, even for tasks that require extended reasoning. This is limiting for open-ended mathematical problem solving, where progress often emerges through intermediate steps, partial arguments, failed attempts, error identification, corrections, and collaboration. To address this gap, we introduce CrowdMath, a dataset of crowdsourced mathematical research discussions from the MIT PRIMES–Art of Problem Solving (AoPS) CrowdMath program (2016–2025), a collaborative research initiative whose discussions have led to peer-reviewed publications. CrowdMath represents these discussions as expert-annotated progress chains that capture not only whether a problem is solved, but also how mathematical progress unfolds across multiple contributors. The dataset enables process-level evaluation through tasks such as post-role classification and next-post prediction, supporting the study of whether models can recognize the role of intermediate contributions and predict plausible next steps in open-ended mathematical discussions, rather than only produce final answers.