Book Overview
Program Evaluation and Randomized Explanatory Trials: A Structural Equation Modeling Approach is written by James Jaccard of New York University. Its focus is on the design and analysis of randomized trials for program evaluation. It expands traditional RCTs by introducing mediation and moderation into them, creating what are called randomized explanatory trials (RETs). RETs seek to explain why a program does or does not work and for what subgroups it works well and what subgroups it does not. Federal agencies have emphasized the importance of incorporating mediation and moderation into RETs for years. However, doing so raises many design and analytic issues that extend well beyond traditional RCTs. This free book covers these issues in ways that are accessible to everyday researchers.
                                                                                                                                                                                                                                                                         
Program Evaluation and Randomized Explanatory Trials
------------------------------------------------------------------------------------------------------------------------------------
SECTION I: FUNDAMENTALS
This section of the book describes conceptual, psychometric, methodological, and statistical fundamentals you need as background to design and analyze RETs. It pulls together diverse literatures that cover the background you need to understand more advanced literatures for RET design and analysis. I assume you have had a basic course in graduate statistics, but not much more. This section covers material that is both non-statistical and statistical in nature and provides a wealth of information relevant to the design and analysis of RETs.
Chapter 1: Randomized Explanatory Trials
This chapter provides an overview of the key theoretical, methodological, and analytic issues that typically must be considered when designing an RET. These include mediator and moderator mapping, the identification of confounders, addressing possible non-linear functional forms, dealing with possible reverse causality, addressing measurement error, evolving logic models for temporal dynamics, and making decisions about sample sizes. I also discuss mixed methods RETs and treating RETs as thought experiments. Finally, I position RETs in the broader context of experimental therapeutics.
Chapter 2: Conceptual Fundamentals for RETs
This chapter addresses three conceptual tasks essential to the design of RETs. First, is the task of carefully analyzing the intervention to define the determinants of the outcome it targets. Second, is the task of placing the targeted mediators into a broader theoretical context that specifies factors outside of the primary logic model that influence the mediators and outcomes. The purpose of this task is to identify confounds and additional variables that need to be taken into account when designing the RET. Third, is the task of defining what constitutes a meaningful effect in an RET, either with reference to the effect of the program on a mediator, the effect of the program on the outcome, or the effect of the mediator on the outcome. The chapter also describes how influence diagrams can assist completion of these three tasks.
Chapter 3: Measurement Fundamentals for RETs
Measurement is at the heart of scientific research and is critical for the analysis of RETs. Chapter 3 considers measurement fundamentals for RETs. It discusses concept-measurement mapping to ensure that measures capture the targeted concepts. It then discusses measurement metrics with a focus on Stevens’s (1951) distinctions between nominal, ordinal, interval and ratio level properties. The chapter addresses the topic of measurement error, including random measurement error and systematic measurement error, followed by a discussion of measurement facets. It describes latent variable representations of measurement and, then measurement invariance, the construction of study specific measures, the practice of dichotomizing measures of continuous constructs, whether to make baseline assessments of outcomes and mediators, and decision making surrounding the frequency and timing of assessments.
Chapter 4: Methodological Fundamentals for RETs
Chapter 4 describes different types of randomized trials, including parallel-group trials, comparative trials, non-inferiority trials, and efficacy versus effectiveness trials. It considers trial designs, including the two-group pretest-posttest design, clustered designs, wait-list designs, crossover designs, and adaptive designs. The concept of a population as contextualized in randomized trials is described and then used to develop the implications of sample imbalance in random assignment. The chapter also describes the mechanics of randomization, phases of randomized trials, demand characteristics and treatment integrity. Finally, the chapter presents and comments on the Consolidated Standards of Reporting Trials (CONSORT) checklist for reporting randomized trials.
Chapter 5: Statistical Fundamentals for RETs: Regression Modeling
Chapter 5 reviews different types of regression modeling that form the foundation of SEM analyses of RETs. These include traditional multiple regression with continuous outcomes, binary regression (including logistic regression, probit regression, log binomial regression, and the modified linear probability model), ordinal regression, multinomial regression, discrete/count regression (including Poisson and negative binomial models, their zero-inflated counterparts, hurdle models, and zero truncated models).
Chapter 6: Statistical Fundamentals for RETs: Advanced Topics
Chapter 6 covers an eclectic set of statistical topics relevant to RETs. It discusses non-linear regression, including smoothers, polynomial regression, and spline regression. It introduces different forms of outlier resistant regression, including quantile regression, trimmed mean regression and MM regression. The chapter discusses ways of dealing with the problem of multiple significance tests through the control of familywise error rates. It considers controversies in invoking such controls, including Bayesian perspectives on the problem. The chapter also considers strategies for constructing margins of errors for the statistics we report, both in the form of confidence intervals and credible intervals. Finally, it underscores the importance of applying sensitivity frameworks when analyzing data and then the topic of reverse causality and instrumental variables.
Chapter 7: Statistical Fundamentals for RETs: Structural Equation Modeling
Chapter 7 introduces you to the basics of structural equation modeling (SEM). The chapter focuses on SEM with continuous mediators and continuous outcomes, but in later chapters, SEM containing nominal, ordinal, binary and count variables are addressed. The chapter outlines traditional SEM in its most fundamental form and is introductory in character.
Chapter 8: Statistical Fundamentals for RETs: Non-Traditional Structural Equation Modeling
Chapter 8 considers less traditional methods for conducting SEM-based analyses. These include Bayesian SEM, limited information SEM (also known as piecewise SEM or reduced form SEM), and Judea Pearl's Structural Causal Model (SCM) approach.
------------------------------------------------------------------------------------------------------------------------------------
SECTION II: MEDIATION ANALYSIS IN RETs
This section of the book covers mediation analysis in the context of RETs.
Chapter 9: Mediation Analysis in RETs: Basic Approaches
This chapter describes traditional approaches to mediation analysis, including the Baron and Kenny method, the coefficient product method, the joint signficance test, Hayes' PROCESS analysis, the MacArthur network model, and causal mediation/SEM.
Chapter 10: Evaluating Effect Sizes in RETs
This chapter considers methods for documenting effect size in mediation analysis. This includes using the variances of disturbance terms, Cohen's d, exceptions to the rule, the number needed to treat, risk differences, relative risks, odds ratios, and omnibus effect size indices. After describing these methods, I discuss how to make judgments about the meanignfulness of effect sizes in RETs.
Chapter 11: Mediation Analysis with Continuous Outcomes
This chapter presents a detailed example for mediation analysis of an RET that has three continuous mediators and a continuous outcome. It lays the foundation for mediation analysis throughout this book. I apply traditional full information SEM to the example, Bayesian SEM, and a variety of limited information estimation (LISEM) frameworks. The latter includes OLS-based LISEM, quantile regression LISEM, robust regression LISEM, Bollen's instrumental variable regression approach, and a form of Bayesian LISEM. I also discuss the causal mediation framework as applied to the example.
Chapter 12: Mediation Analysis with Binary Outcomes
This chapter presents a detailed example for mediation analysis of an RET that has a binary outcome. It discusses little recognized assumptions of logistic and probit regression when conducting mediation analysis. It then walks you through both limited information SEM as applied to a numerical example and full information SEM. The chapter considers the modified linear probability model, probit regression, and Bayesian modeling. I provide a ratioanle for preferring probit-based modeling to logistic modeling. I also discuss the causal mediation framework.
Chapter 13: Mediation Analysis with Ordinal and Nominal Outcomes
This chapter considers how to analyze data from RETs when the outcome variable is ordinal or nominal. When the outcome is ordinal, I discuss two approaches to analysis. The first is a latent response approach that models the continuous latent response, y*, thought to underlie the ordinal measure. In this case, the ordinal measure is viewed as a crude indicator of the underlying continuous dimension of interest, with the latter being of primary interest. The second approach, called the probability approach, is to model the ordinal measure directly, not its underlying continuous dimension of symptom improvement. In this case, interest is with the categories of the scale in their own right and statements about how the program affects the proportion of people in the different scale categories. The chapter also considers the case where the outcome variable is nominal in character and models RET effects using multinomial logistic regression formulations.
Chapter 14: Mediation Analysis with Count Outcomes
To be added
Chapter 15: Non-Linear and Specialized Modeling in Mediation Analysis
This chapter focuses on non-linear modeling of mediation. Traditional mediation analysis tends to work with linear models. The current chapter expands your statistical tool box by considering in a clear and understandable way the topics of polynomial regression, spline regression, traditional non-linear regression, Bayes additive regression trees, generalized additive models, recursuve partitioning (CART) models, cluster analysis, and latent profile/class analysis. Applications of each of these techniques to the analysis of RETs are discussed.
Chapter 16: RETs with Follow-Ups: Longitudinal Modeling
To be added
Chapter 17: Mediator Relative Importance and Exploratory Mediator Analysis
This chapter describes methods for evaluating the relative importance of mediators in RETs. Two topics are central. First, given a set of mediators, one may want to order them in terms of their relative importance in influencing the outcome. The idea is that if limited resources demand we focus activities on a smaller subset of mediators, then knowing the relative importance of mediators in shaping an outcome will help us choose mediators to prioritize. Second, the chapter considers the case where one has identified and measured so many program mediators that one must apply data reduction strategies to reduce the number of mediators to a workable set. Topics include lasso regression, generalized additive models, all possible regressions, and dominance analysis, as well as approaches from data mining.
------------------------------------------------------------------------------------------------------------------------------------
SECTION III: MODERATION ANALYSIS IN RETs
To be added
Chapter 18: Introduction to Moderation Analysis
This chapter provides an overview of moderation and ways of parameterizing it in RETs. It discusses correlational methods using change scores that some researchers use to identify moderators and shows the pitfalls of such strategies. It discusses the conceptual difference between interaction effects and moderation effects and the ways researchers think of their respective parameterizations. The essence of moderation analysis is moderation contrasts. The chapter describes how to quantify such contrasts for nominal and continuous variables. It considers the interesting case when X moderates itself in the X-Y relationship and methods for graphing moderated relationships. The importance of distinguishing between ordinal and disordinal effects is considered. Finally, issues for asserting group equivalence in causal coefficients are briefly considered. The chapter serves as an overview for more advanced topics considered in this section of the book.
Chaoter 19: Moderation Analysis with Product Terms
To be added
Chapter 20: Moderation Analysis with Multiple Group SEM
To be added
Chapter 21: Exploratory Moderation Analysis and Non-linear Dynamics
To be added
Chapter 22: Moderated Mediation
To be added
Chapter 23: Mediated Moderation
To be added
Chapter 24: Moderated Moderation
To be added
------------------------------------------------------------------------------------------------------------------------------------
SECTION IV: ADDITIONAL ISSUES IN THE ANALYSIS OF RETs
To be added
Chapter 25: Group Adminstered Interventions and Clustered Designs
This chapter considers the analysis of RETs in which groups (e.g., classrooms, schools, clinics) rather than iindividuals are randomly assigned to the interevention and control conditions. These are often called cluster randomized designs; they introduce analytic challenges because of dependencies that can occur between members of the same group (e.g., the disruptive behavior of one group member can affect the other members in the group). The present chapter describes analytic methods for addressing these complications. I consider analytic strategies that treat the clustering as nuisance variables as well as strategies that seek to understand cluster characteristics that impact intervention effectiveness. The latter include multi-level structural equation modeling methods, including methods to accomodate cases where there a small number of clusters in a study.
Chapter 26: Missing Data
This chapter covers how to deal with missing data when analyzing RETs. It does NOT cover how to deal with treatment dropouts as a source of missing data (see Chapter 27). It instead addresses missing data for questions people choose purposely or inadvertently not to answer during non-treatment sessions. I describe the classic mechanisms for missing data including missing data completely at random (MCAR), missing data at random (MAR), and data not missing at random (NMAR). Modern methods for dealing with these mechanisms are described with an emphasis on full information maximum likelihood (FIML) and multiple imputation. The statistical literature on these topics is vast and confusing, with significant advances being made with each passing year. The chapter provides a coherent overview of the literature and makes recommendations for best analytic practices in the context of RETs
Chapter 27: Intent to Treat and Per Protocol Modeling
This chapter considers modern methods for per protocol and intent to treat analyses in randomized trials. It first builds a case for the importance of efficacy (per protocol) based analyses and seeks to debunk many of the biases against them in favor of intent to treat analyses. Methods for proper per protocol analysis are then reviewed, including the direct covariate approach, IPTW weighting approaches, complier average causal effect and instrumental variable approaches, G computation, and targeted maximum likelihood estimation. The chapter then describes issues in the conduct of intent to treat analyses and ways of dealing with missing data due to treatment dropouts.
Chapter 28: Sample Size Considerations
This chapter addresses sample size issues when designing RETs. It describes the central role of expected sampling error when making sample size decisions and factors that affect sampling error. Sample size not only affects statistical power but it also can affect asymptotic theory, parameter bias, robustness, confidence interval coverage, margins of error, and covariance stability. The chapter elucidates these impacts. The chapter describes how to increase statistical power not only by increasing sample size but also by strategically introducing covariates into your model. It disucsses how to conduct power analysis for a wide range of statistical methods, both traditional and in the context of SEM. It shows you how to conduct Monte Carlo simulations to help you make sample size decisions. Finally, it reviews various small sample statistical methods that can be used to analyze RETs when your sample size is limited.
Chapter 29: Factorial and Dismantling Designs, Multi-Treatment RETs
To be added
Chapter 30: Epilogue
To be added
------------------------------------------------------------------------------------------------------------------------------------
This book is introductory and written for everyday researchers who are not familiar with the ins and outs of SEM but who are knowledgeable about the basics of multiple regression and who are interested in program evaluation using randomized trials. Those with advanced knowledge of SEM may be disappointed with the level of coverage, but I believe there is a gap relative to the type of book I have written. I place this book at the same introductory level as Kline's introduction to SEM, perhaps a tad more advanced, but with me having the freedom to expand on topics given the webbook format (publishers invariably tie one's hands with page constraints). The book is far more than a book on SEM. It considers many facets of RET design.
The book describes how to analyze RETs using structural equation modeling (SEM). It works primarily with Mplus software but also, via this website, other R and SEM programs (e.g., lavaan, MIIVsem). The website also provides "programs" that you can run to conduct a wide range of analyses relevant to SEM as applied to RETs. Most everything you need to analyze your RET data is available in Mplus or here. 
Below is a brief overview of each chapter. To see the description, click below on the triangle to the left of the chapter title.
This website has many supplemental materials. The 'Syntax' tab covers the basics of Mplus and lavaan syntax. The 'Import/Syntax' tab imports data from a wide range of software programs (e.g., SPSS, Excel, SAS, STATA) and creates data files for use in Mplus and R. It also has a syntax generator that writes program code for you to make programming in Mplus and lavaan easier. The 'Output' tab presents example Mplus output and provides 'tool tips' for entries to help you interpret the output. The 'Programs' tab provides over 100 programs you can run in R to analyze your data using techniques discussed in the book (you do not need to know R programming to use the programs - if you can copy and paste, you can use the programs). The 'Simulations' tab provides point-and-click interactive simulations you can conduct to gain perspectives on sampling error, power, and Type I errors. The 'Resources' tab provides extensive supplemental materials and links around the web for each chapter.