Cause, Effect, and the Structure of the Social World

tags

Causal Inference

Notes

a central empirical claim: most reforms and interventions in the criminal legal space are shown to have little lasting impact when evaluated with gold-standard methods of causal inference.

NOTER_PAGE: (3 0.16606498194945848 . 0.45514018691588787)

change is hard to engineer.

NOTER_PAGE: (3 0.3299638989169675 . 0.2672897196261682)

stabilizing forces usually steer them back onto the path they would have been on absent the intervention.

NOTER_PAGE: (3 0.3667870036101083 . 0.3186915887850467)

That’s not to say that the social world is static: to the contrary, it’s changing all the time. This is a claim about the ability to engineer such change using a particular type of intervention.

NOTER_PAGE: (3 0.43249097472924186 . 0.24953271028037385)

Hoping to replicate the medical-context success, the social sciences have embraced RCTs.

NOTER_PAGE: (3 0.5292418772563177 . 0.2280373831775701)

You can’t pilot test systemic reform

NOTER_PAGE: (5 0.2671480144404332 . 0.2280373831775701)

those who hope to achieve change can either: (1) focus on interventions with immediate and direct benefit; (2) continue with limited-scope interventions in the hopes that they have the type of benefit that is difficult to measure; or (3) seek systemic reform, with all its uncertainties.

NOTER_PAGE: (5 0.4180505415162455 . 0.3149532710280374)

Martinson made the infamous proclamation that “nothing works” when it comes to prisoner rehabilitation,

NOTER_PAGE: (6 0.251985559566787 . 0.4523364485981309)

failure of social programs was so ubiquitous that it should be known as the Iron Law of Evaluation.

NOTER_PAGE: (6 0.2931407942238267 . 0.21869158878504674)

I. WHAT IS AN RCT? WHY IS IT POWERFUL?

NOTER_PAGE: (7 0.1927797833935018 . 0.34205607476635513)

You can’t control for motivation. You can’t fully control for access to resources.

NOTER_PAGE: (10 0.6028880866425993 . 0.5102803738317757)

Publication Bias

NOTER_PAGE: (11 0.19927797833935018 . 0.26542056074766357)

RCTs provide evidence about the causal impact of interventions on outcomes for a particular sample.

NOTER_PAGE: (15 0.3494584837545126 . 0.44485981308411215)

interventions evaluated by RCTs need to be manipulable, meaning that they need to be something that can be randomly assigned

NOTER_PAGE: (15 0.46498194945848376 . 0.2822429906542056)

interventions need to be isolable,

NOTER_PAGE: (15 0.4931407942238267 . 0.28504672897196265)

Socioeconomic class, for instance, is not isolable.

NOTER_PAGE: (16 0.14296028880866427 . 0.3271028037383178)

Interventions also tend to be of limited scope.

NOTER_PAGE: (16 0.24693140794223825 . 0.2411214953271028)

experiments are likely to be considered unethical if they make an individual appreciably worse off.

NOTER_PAGE: (16 0.251985559566787 . 0.6242990654205608)

it is rare to find groups that vary massively along one dimension but are otherwise statistically identical.

NOTER_PAGE: (16 0.3588447653429603 . 0.39252336448598135)

researchers tend to be interested in longer-term or indirect effects of a program.

NOTER_PAGE: (17 0.20288808664259927 . 0.2953271028037383)

not always clear whether evidence from one particular sample provides useful information about other times, places, and groups of people.

NOTER_PAGE: (17 0.2945848375451263 . 0.6084112149532711)

the shift toward using RCTs to evaluate social policies has meant prioritizing small-bore questions over more important (and more difficult to answer) ones.

NOTER_PAGE: (18 0.16101083032490973 . 0.5355140186915888)

The interventions evaluated by RCTs may be of limited scope, but they are actionable interventions.

NOTER_PAGE: (19 0.17978339350180506 . 0.3850467289719626)

questions answered by RCTs are directly relevant to the activist, NGO, or policymaker who wants to know what they can do to change the world.

NOTER_PAGE: (19 0.27292418772563176 . 0.2252336448598131)

II. FIFTY-PLUS YEARS OF RCT EVIDENCE

NOTER_PAGE: (19 0.5061371841155234 . 0.35233644859813085)

most reforms and interventions in the criminal legal space have little to no lasting effect when evaluated by RCTs, and the occasional success usually fails to replicate when evaluated in other settings.

NOTER_PAGE: (19 0.5270758122743682 . 0.4177570093457944)

only 29 of the 122 studies (24%) found statistically significant impacts in the desired direction.111 Furthermore, the estimated treatment effect was in the desired direction in only 77 out of 122 studies (63%).

NOTER_PAGE: (21 0.1963898916967509 . 0.305607476635514)

study found that an initial arrest led to substantially fewer repeat incidents over the subsequent six months.

NOTER_PAGE: (22 0.15018050541516245 . 0.33644859813084116)

This study was enormously influential and was used to support mandatory arrest laws across the nation.

NOTER_PAGE: (22 0.21660649819494585 . 0.24485981308411217)

Combining data across all seven studies, researchers later concluded that arrest did not, in fact, have a consistent or large effect on recidivism.

NOTER_PAGE: (22 0.2787003610108303 . 0.34392523364485983)

The original study and three follow-up studies (all RCTs) showed large reductions in recidivism

NOTER_PAGE: (22 0.35018050541516244 . 0.6495327102803738)

Once they adjusted for publication bias, MST’s effect size was close to zero

NOTER_PAGE: (23 0.2187725631768953 . 0.6168224299065421)

within a few years, “swift, certain, and fair” sanctioning had been adopted in at least 160 instances.

NOTER_PAGE: (24 0.20938628158844763 . 0.2775700934579439)

Again, the National Institute of Justice funded RCTs to try and replicate Project HOPE’s success across five sites.137 The results were not promising:

NOTER_PAGE: (24 0.23321299638989168 . 0.28037383177570097)

teaching youths to “think slow”

NOTER_PAGE: (25 0.20216606498194944 . 0.3504672897196262)

effects for subsequent cohorts were close to zero and statistically insignificant.

NOTER_PAGE: (26 0.14296028880866427 . 0.5214953271028038)

While the original success was published in economics’ most prestigious journal and received widespread media attention, the subsequent failure to replicate is mentioned only tangentially in the back pages of an unpublished working paper on a different topic.

NOTER_PAGE: (26 0.19999999999999998 . 0.33271028037383177)

the interventions which get evaluated across multiple sites are those that, based on theory or prior research, are believed to be particularly promising.

NOTER_PAGE: (28 0.17689530685920576 . 0.31401869158878504)

the most widely studied strategies in criminal justice seem to have, at most, small and contested effects.

NOTER_PAGE: (29 0.25415162454873647 . 0.41401869158878507)

the one area of causal inference research in which lasting or replicable effects are found somewhat more frequently is interventions made to the lives of youths.

NOTER_PAGE: (29 0.28447653429602887 . 0.3037383177570094)

III. THE STRUCTURE OF THE SOCIAL WORLD

NOTER_PAGE: (31 0.22382671480144403 . 0.3457943925233645)

A. Stabilizers, Cascades, and Complexity

NOTER_PAGE: (31 0.423826714801444 . 0.22429906542056074)

the social world is full of what I call “stabilizers” and short on what I refer to as “cascades.”

NOTER_PAGE: (31 0.5184115523465703 . 0.2822429906542056)

Stabilizers are the set of socioeconomic forces that resist externally-imposed change.

NOTER_PAGE: (31 0.5891696750902526 . 0.24672897196261684)

whatever socioeconomic forces made it hard for that individual to find a job in the first place—for example, a society in which access to opportunity is deeply segregated—prove to be powerful inhibitors. After the program is over, the participant returns to the place they would have been absent the intervention.

NOTER_PAGE: (31 0.6541516245487364 . 0.22897196261682243)

Cascades are forces that magnify small changes, that turn a small intervention into a large and lasting effect.

NOTER_PAGE: (32 0.31263537906137184 . 0.31401869158878504)

Cascade narratives can be very compelling. But that doesn’t mean they are true. RCTs teach us that very few interventions launch such a virtuous cycle of accumulating benefits. The social scientists’ holy grail—the small, inexpensive intervention with large, widespread, and lasting gains—appears to be mostly myth.

NOTER_PAGE: (32 0.47003610108303245 . 0.2457943925233645)

The original success could be due to the charisma of a key figure,

NOTER_PAGE: (32 0.6274368231046931 . 0.43457943925233644)

B. Scope of the Claim

NOTER_PAGE: (33 0.22815884476534296 . 0.22897196261682243)

1. Does the Claim Apply Outside of the Criminal Legal Space?

NOTER_PAGE: (33 0.4592057761732852 . 0.24205607476635516)

Perhaps there is something unique about the people studied by RCTs in the criminal legal space that could limit generalizability.

NOTER_PAGE: (33 0.4823104693140794 . 0.24392523364485982)

Microcredit (loaning small amounts of money, often to women) was, for many years, the darling of the development world. Eventually, it was shown to have little to no net benefit in most places.

NOTER_PAGE: (34 0.5682310469314079 . 0.34205607476635513)

Health insurance, randomly allocated via lottery, was shown to increase healthcare usage and reduce bills sent to a collections agency.176 Yet, it had no statistically significant effect on physical health or labor market outcomes.

NOTER_PAGE: (34 0.5978339350180505 . 0.6009345794392523)

the Iron Law of Evaluation: “The expected value of any net impact assessment of any large-scale social program is zero.”

NOTER_PAGE: (35 0.1407942238267148 . 0.3186915887850467)

when it came to more indirect outcomes—health, consumption habits, etc.—the researchers found that, overall, “the lives of recipients were not altered dramatically by the payments offered in the experiments.”

NOTER_PAGE: (35 0.2967509025270758 . 0.22429906542056074)

The more speculative it sounds, the less likely it is there will be a robust and replicable causal relationship.

NOTER_PAGE: (35 0.38700361010830325 . 0.3224299065420561)

2. Does the Claim Apply Beyond the Set of Questions Answered and Answerable by RCTs?

NOTER_PAGE: (35 0.43249097472924186 . 0.2514018691588785)

RCTs tend to focus on questions that aren’t a priori obvious.

NOTER_PAGE: (35 0.49675090252707577 . 0.24299065420560748)

There really is no good way to empirically identify the impact of a five-year prison sentence versus no criminal justice involvement at all.

NOTER_PAGE: (36 0.47725631768953064 . 0.3990654205607477)

interventions evaluable via rigorous method of empirical causal inference.

NOTER_PAGE: (37 0.1862815884476534 . 0.33925233644859815)

IV. IMPLICATIONS

NOTER_PAGE: (37 0.5169675090252708 . 0.43644859813084114)

A. Myth

NOTER_PAGE: (38 0.18267148014440432 . 0.23644859813084113)

Under the engineer’s view, the causal structure of the social world can be mapped using RCTs and other scientific methods, and, once mapped, it can be manipulated to achieve social goals. Certain interventions yield such consistent and replicable success that they can be labeled “best practices.” And meaningful reform can be achieved with reduced risk and uncertainty because the interventions have been rigorously evaluated before scaling up.

NOTER_PAGE: (38 0.22527075812274366 . 0.2906542056074766)

emphasis on using RCTs to identify “what works,”

NOTER_PAGE: (39 0.17545126353790613 . 0.34953271028037386)

The empirical research that gets celebrated is that which purports to successfully map some quadrant of the causal machine, that shows how an intervention successfully changes an important outcome.

NOTER_PAGE: (40 0.28736462093862813 . 0.40560747663551405)

B. Social Change

NOTER_PAGE: (40 0.6693140794223826 . 0.22897196261682243)

a focus on RCTs means a focus on a type of reform that is inherently conservative.

NOTER_PAGE: (42 0.15812274368231047 . 0.3747663551401869)

RCTs require holding all else constant except the treatment. This “limits the focus to interventions that leave systems intact and change some element that is manipulable without doing ‘damage’ to the system.”

NOTER_PAGE: (42 0.20216606498194944 . 0.2355140186915888)

If limited-scope changes to one component of a complex system rarely have much lasting effect, then this leaves us with only a few options: (1) focus on interventions with immediate, direct net benefits; (2) continue engaging in limited-scope interventions with an acceptance that they are likely to have only a limited impact; or (3) try to implement change that goes beyond what is evaluable with RCTs.

NOTER_PAGE: (42 0.38844765342960286 . 0.5588785046728972)

(1) Give a man a fish:

NOTER_PAGE: (42 0.5061371841155234 . 0.2457943925233645)

there remains a straightforward and obvious way to ameliorate harm: simply give people what they need.

NOTER_PAGE: (42 0.5646209386281588 . 0.2841121495327103)

(2) Uncertain incrementalism:

NOTER_PAGE: (43 0.1444043321299639 . 0.24485981308411217)

(3) Systemic reform:

NOTER_PAGE: (43 0.4274368231046931 . 0.24672897196261684)

C. What Is the Structure of the Social World?

NOTER_PAGE: (43 0.6895306859205775 . 0.2280373831775701)

D. How Should We Learn About How To Achieve Desired Change?

NOTER_PAGE: (44 0.4267148014440433 . 0.2327102803738318)

E. On Research and Knowledge Generation

NOTER_PAGE: (45 0.6534296028880866 . 0.22990654205607478)

the engineer’s view is extraordinarily appealing.

NOTER_PAGE: (46 0.23826714801444043 . 0.2925233644859813)

the people best positioned to dispel the myth are those who stand to lose the most from its absence.

NOTER_PAGE: (46 0.32563176895306856 . 0.30934579439252335)

many researchers don’t think of the engineer’s view as a myth.

NOTER_PAGE: (46 0.4462093862815884 . 0.2953271028037383)

CONCLUSION

NOTER_PAGE: (47 0.21444043321299638 . 0.4532710280373832)

The fact that outside forces—interventions— are largely unsuccessful at engineering change in people’s lives does not necessarily mean that humans are powerless beings in the throes of social forces. Rather, it suggests that people have already fought to create the best lives they could for themselves given the circumstances.

NOTER_PAGE: (47 0.31263537906137184 . 0.4654205607476636)

the constraints that remain appear to be deep, structural, and hard to shift. That doesn’t mean they are immovable, but just that they usually aren’t moveable with the type of intervention evaluable via RCT.

NOTER_PAGE: (47 0.43610108303249095 . 0.3682242990654206)

As for how to move them—I don’t know. Moreover, I don’t think we can know, or at least not with the high levels of confidence promised by the engineer’s view. We will proceed, but must do so with the humility of uncertainty.

NOTER_PAGE: (47 0.4714801444043321 . 0.6242990654205608)