Interpretable Model Learning and Adaptive EM Algorithm for Poisson Mixture Models

Abstract

This dissertation consists of two parts: (a) interpretable model learning, and (b) partialExpectation-Maximization (PEM) algorithm for fitting Poisson mixtures, aimed to address challenges in modeling complex data, and analyzing large or online data in batches, respectively. Statistical modeling is a key step for supporting evidence-based decision-making. In the first part of this dissertation, we developed a comprehensive learning strategy for constructing interpretable statistical models for data with multiple covariates x e and outcome y. Linear models are easy to interpret but limited in the flexibility of modeling nonlinear relations in data, whereas deep neural networks and nonparametric models offer some flexibility at the cost of interpretability. Our strategy is to develop a semiparametric model learning approach —incorporating linear relation of some covariates (denoted by x1) with y, nonlinear relation of some other covariates (denoted by x2) with y, and nonparametric relation of the remaining covariates (denoted by x3) with y. The nonlinear relation here is meant to depend on a few unknown parameters and its shape to be determined by data, and hence is also termed as seeking a nonlinear transformation of x2. This model will be called “S-Interpretable” model, bridging the gap between the fully linear and fully nonparametric models. The challenges to build our “S-Interpretable” models are: (1) properly partitioning covariates x into x1, x2, and x3, (2) determining suitable transformations for x2 when nonlinear effects can be approximated by unknown parametric functions, and (3) jointly estimating linear, nonlinear, and nonparametric components. To address these challenges, we developed a data-driven pipeline that (1) determines x1,x2,x3 based on the data (with an option for human input), (2) suggests transformation forms for x2, (3) enables simultaneous estimation of all components, (4) validates transformation sufficiency, and (5) evaluates overall model performance. Our contributions are not only in developing this interpretable model learning strategy, but also developing (i) A support for transformation necessity and sufficiency using a semiparametric bootstrap method based on effective degrees of freedom (EDF), (ii) A simultaneous estimation procedure allowing transformation terms to follow change-point parametric functions, (iii) Comprehensive model evaluation strategies, including repeated cross-validation and crossconformal prediction sets, and (iv) An application to a real-world study of Tibetan women reproductive success. These developments are not only of interest for each of the four components (i)-(iv) but also contribute to the system science that requires an ensemble for data science, such as our “S-Interpretable” model learning. In the second part of this dissertation, we proposed an adaptive partial EM algorithm for estimating Poisson mixture models to analyze heterogeneous count data. The full EM algorithms require processing the full dataset in each iteration, posing efficiency and scalability issues when having large data. Sun et al. (2025+) developed a partial EM (PEM) estimation method for a Gaussian mixture model, which is competitive with other methods in theory and computational efficiency. In this dissertation, we follow their idea and extend it to the Poisson mixture case. We derive the partial maximum likelihood estimators (MLE) for Poisson mixture parameters for data composed of the previous estimates and a new batch of data. We handle cases with and without an extra component in the new data batch to those from the previous estimates. Our method demonstrates significant improvements in computational efficiency and scalability compared to existing benchmarks. Theoretical guarantees are provided for the convergence of the algorithm and the consistency of the MLE.