Addressing nonignorable attrition with double sampling and bounds: the estimators of Coppock, Gerber, Green, and Kern (2017), Political Analysis 25(2):188-206, for randomized experiments in which some outcomes are missing.
Install the released version from CRAN:
install.packages("attrition")Or the development version from GitHub:
# install.packages("remotes")
remotes::install_github("acoppock/attrition")Attrition occurs when some subjects do not report outcomes or measurements are not recorded for some units. Attrition can happen in many settings, including:
Under attrition, naive estimators are biased for the average
treatment effect. Chapter 7 of Gerber and Green (2012) gives a textbook
introduction to the problem and to the bounding approaches implemented
in the attrition package. Coppock, Gerber, Green, and Kern
(2017) describe a double sampling design for narrowing the bounds.
| Function | What it assumes |
|---|---|
estimator_ev() |
The outcome has known lower and upper limits (Manski 1990). |
estimator_ds() |
The same, plus a random follow-up sample of nonrespondents. |
estimator_ds_sens() |
The same, with ignorability allowed to fail for a fraction
delta of the follow-up nonrespondents. |
sensitivity_ds() |
A search over delta for the point where the interval
starts to include zero. |
estimator_trim() |
Treatment moves response in one direction only (Lee 2009), or nothing beyond random assignment (Imai 2008). The outcome need not be bounded. |
The first four take a strata argument for
poststratification on a discrete covariate, which targets the same
identification region and estimates it more precisely;
estimator_trim() does not. All five have
tidy() methods and a formula interface for use with DeclareDesign.
The package ships the replication study from the paper as
levendusky_replication: a two-wave survey experiment in
which 1,980 subjects were asked about perceived polarization, and 536 of
them did not answer the second wave.
Refusing any assumption about the missing outcomes gives worst-case bounds. Because the outcome runs from 0 to 6, filling every missing value in one group with 0 and the other with 6 gives the lowest effect the data can support, and reversing the fills gives the highest.
library(attrition)
fit_ev <- estimator_ev(Y = Y_polarization_w2,
Z = Z,
R = R1,
minY = 0,
maxY = 6,
data = levendusky_replication)
tidy(fit_ev)
#> # A tibble: 3 × 10
#> term estimate std.error conf.low conf.high estimate_lower estimate_upper
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 bounds NA NA -1.67 1.84 -1.54 1.71
#> 2 lower_bou… -1.54 0.0790 NA NA NA NA
#> 3 upper_bou… 1.71 0.0767 NA NA NA NA
#> # ℹ 3 more variables: std.error_lower <dbl>, std.error_upper <dbl>,
#> # outcome <chr>The effect lies somewhere between -1.54 and 1.71, which is honest and nearly useless. Little of that width is sampling error: the confidence interval is barely wider than the bounds. A larger sample would not have helped, because the problem is 536 unknown outcomes rather than noise.
Double sampling addresses the unknown outcomes directly. After the first round of data collection, a random sample of the nonrespondents is drawn and pursued with more effort than the first attempt. Here, 50 nonrespondents per condition were offered $4.00 rather than the original $1.00, and 72 of those 100 answered. Because they are a random sample of the nonrespondents, their outcomes stand in for all 536, and only the 28 who refused twice still need worst-case treatment.
fit_ds <- estimator_ds(Y = Y_polarization_w2,
Z = Z,
R1 = R1,
Attempt = Attempt,
R2 = R2,
minY = 0,
maxY = 6,
data = levendusky_replication)
tidy(fit_ds)
#> # A tibble: 3 × 10
#> term estimate std.error conf.low conf.high estimate_lower estimate_upper
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 bounds NA NA -0.528 0.745 -0.342 0.572
#> 2 lower_bou… -0.342 0.113 NA NA NA NA
#> 3 upper_bou… 0.572 0.105 NA NA NA NA
#> # ℹ 3 more variables: std.error_lower <dbl>, std.error_upper <dbl>,
#> # outcome <chr>The identification region shrinks by a factor of 3.6, from 3.25 points wide to 0.91.
vignette("attrition") works through the design and all
five estimators on the shipped data, reproducing the paper’s Table 3
along the way. It also draws the imputation that
estimator_ev() averages over and checks the picture against
the estimates.
attrition 1.0.0 was prepared by Alexander Coppock working with Claude
(Anthropic), across the rewrite of the estimators, the test suite, and
the documentation. The method and the original implementation come from
Coppock, Gerber, Green, and Kern (2017). The 1.0.0 release rewrote the
internals, added a formula interface and tidy() methods,
corrected several defects in the earlier code, and wrote the
vignette.
The code base has been reviewed but not written line by line, so the
guarantee offered is not that every line has been vouched for. It is
that the package reproduces the published analysis it implements.
vignette("attrition") reproduces Table 3 of the paper, and
the test suite holds each estimator to the published quantities, which
have not changed. The analytic variance added to
estimator_trim() was checked three ways: against the
algebraically distinct form in Tauchmann’s Stata leebounds,
against the Monte Carlo standard deviation of the estimator across
sample sizes from 1,000 to 64,000, and by the coverage of a conventional
interval around each bound endpoint.
Coppock, Alexander, Alan S. Gerber, Donald P. Green, and Holger L. Kern (2017). Combining Double Sampling and Bounds to Address Nonignorable Missing Outcomes in Randomized Experiments. Political Analysis 25(2):188-206. https://doi.org/10.1017/pan.2016.6
Gerber, Alan S., and Donald P. Green (2012). Field Experiments: Design, Analysis, and Interpretation. New York: W. W. Norton.
Imbens, Guido W., and Charles F. Manski (2004). Confidence Intervals for Partially Identified Parameters. Econometrica 72(6):1845-1857. https://doi.org/10.1111/j.1468-0262.2004.00555.x
Lee, David S. (2009). Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects. Review of Economic Studies 76(3):1071-1102. https://doi.org/10.1111/j.1467-937X.2009.00536.x
Manski, Charles F. (1990). Nonparametric Bounds on Treatment Effects. American Economic Review Papers and Proceedings 80(2):319-323.