- tags
- Predictive Analytics Digisprudence
Notes
INTRODUCTION
NOTER_PAGE: (4 0.11333750782717594 . 0.3897893030794165)
Algorithmic predictions are significantly different from other types of algorithmic inferences because they involve the element of time. The temporal dimension dramatically changes the implications
NOTER_PAGE: (6 0.18722604884157795 . 0.38816855753646673)
we use the term “prediction” more precisely to refer to a type of inference that involves forecasting future events that can’t be verified in the present,
NOTER_PAGE: (6 0.3318722604884158 . 0.5210696920583469)
matters asserted in predictions are presently unvested and contingent.
NOTER_PAGE: (6 0.3926111458985598 . 0.3897893030794165)
The problems stem from dealing with a probable (or possible) but uncertain future
NOTER_PAGE: (7 0.32748904195366313 . 0.4837925445705024)
Algorithmic predictions not only forecast the future but also have the power to create and control it.
NOTER_PAGE: (7 0.43894802755165935 . 0.2025931928687196)
Privacy law is built around a true-false dichotomy and provides rights of correction and duties to maintain accurate records. But predictions are neither true nor false and don’t fit into this dichotomy.
NOTER_PAGE: (8 0.2949279899812148 . 0.3038897893030794)
I. UNDERSTANDING ALGORITHMIC PREDICTIONS
NOTER_PAGE: (9 0.11208515967438948 . 0.25364667747163694)
A. INFERENCES, PROFILING, AND PREDICTIONS
NOTER_PAGE: (9 0.44270507201001874 . 0.2633711507293355)
B. PAST OR PRESENT VS. FUTURE
NOTER_PAGE: (11 0.4871634314339386 . 0.33144246353322526)
established methods of proof and evidence. The legal system has developed elaborate rules and procedures for adjudicating the past.
NOTER_PAGE: (11 0.6512210394489668 . 0.19692058346839544)
no comparable legal architecture for litigating the future.
NOTER_PAGE: (11 0.7319974953036944 . 0.38006482982171796)
making the temporal distinction and singling out predictions can be difficult
NOTER_PAGE: (12 0.1415153412648716 . 0.6385737439222042)
If one were to deny a person a job or a loan and claim that it is based on a current heart disease, we would characterize the decision likely to be based on a prediction
NOTER_PAGE: (12 0.3926111458985598 . 0.6191247974068071)
C. THE DRAMATIC RISE OF ALGORITHMIC PREDICTIONS
NOTER_PAGE: (12 0.7739511584220413 . 0.2139384116693679)
Credit scoring has long been beset with problems, such as lack of transparency, inaccuracy, bias, and unfairness.
NOTER_PAGE: (13 0.5604257983719474 . 0.2106969205834684)
Notwithstanding the problems with credit scoring, its use skyrocketed as well as spread to a wide array of decisions beyond credit. Credit scoring grew in part because it was fast, cheap, and consistent.37 Today, as Oscar Gandy notes, the “use of credit scores has expanded well beyond its initial applications, finding extensive use in housing, insurance, residential services, and employment decisions.”
NOTER_PAGE: (13 0.641202254226675 . 0.19611021069692058)
credit scoring is shifting from a reliance on a “few variables” and “human discretion” to using a broader array of personal data and machine learning.
NOTER_PAGE: (14 0.12273011897307451 . 0.5170178282009724)
“Predictive technologies are spreading through the criminal justice system like wildfire.”
NOTER_PAGE: (14 0.5829680651221039 . 0.38006482982171796)
Henry Ford took Taylorism in a new more totalitarian direction, creating a “Sociological Department” that spied on his workers’ private lives. His system was not primarily about predicting worker success; it was to “create model people.”
NOTER_PAGE: (16 0.24733876017532874 . 0.6661264181523501)
NOTER_PAGE: (18 0.39010644959298685 . 0.33306320907617504)
D. THE LIMITATIONS OF ALGORITHMIC PREDICTIONS
NOTER_PAGE: (21 0.1696931747025673 . 0.23014586709886548)
algorithmic predictions rest on certain assumptions which are not ineluctably true. These assumptions are that (1) the past repeats itself and thus the future will be similar to the past; (2) an individual will continue to say and do things similarly as in the past; and (3) groups of individuals sharing similar characteristics or traits act similarly.
NOTER_PAGE: (21 0.26988102692548527 . 0.25202593192868716)
algorithmic predictions are never 100% accurate because the future is never 100% certain.
NOTER_PAGE: (21 0.4120225422667501 . 0.3589951377633711)
NOTER_PAGE: (21 0.725735754539762 . 0.3038897893030794)
NOTER_PAGE: (22 0.10832811521603006 . 0.3103727714748784)
II. THE PROBLEMS WITH ALGORITHMIC PREDICTIONS
NOTER_PAGE: (22 0.4483406386975579 . 0.3306320907617504)
A. THE FOSSILIZATION PROBLEM
Algorithmic predictions lead to what we call the “fossilization problem” – they can reify the past and make it dictate the future.
NOTER_PAGE: (23 0.4871634314339386 . 0.20097244732576985)
treating individuals unequally based on their past behaviors or conditions with an assumption that their past is likely to repeat
NOTER_PAGE: (23 0.5222291797119599 . 0.36304700162074555)
Algorithmic predictions often weigh prior data too heavily.
NOTER_PAGE: (23 0.628052598622417 . 0.26499189627228525)
There is a value in not tethering people to their past.
NOTER_PAGE: (24 0.3112085159674389 . 0.20502431118314424)
Algorithmic predictions shackle people not just to their own past but also to the past of others.
NOTER_PAGE: (24 0.5623043206011271 . 0.23905996758508913)
Discrimination and bias are so marbled throughout past data that they cannot readily be extricated.
NOTER_PAGE: (24 0.6593613024420789 . 0.3273905996758509)
Inequality hangs over the past like fog.
NOTER_PAGE: (25 0.21415153412648716 . 0.20097244732576985)
Advantages become etched into the future because they exist in the data. With fossilization, the losers keep losing and the winners keep winning.
NOTER_PAGE: (25 0.3318722604884158 . 0.2106969205834684)
Algorithmic predictions answer the question of “what will happen in the future?’ by asking “what happened in the past?” In this way, the past will always cast a shadow on the future.
NOTER_PAGE: (25 0.4013775829680651 . 0.35170178282009723)
police departments have gathered copious data about traffic control and accidents “to satisfy the requirements of insurance companies” but “rarely collected the kinds of data that would support an analysis of racial bias in traffic stops, searches, and arrests.”
NOTER_PAGE: (25 0.6944270507201001 . 0.2025931928687196)
“Every action—or refusal to act—on the part of a police officer, and every similar decision made by a police department, is also a decision about how and whether to generate data.”
NOTER_PAGE: (26 0.18346900438321853 . 0.28038897893030795)
Was he free? Was he happy? The question is absurd: Had anything been wrong, we should certainly have heard.
NOTER_PAGE: (26 0.43644333124608636 . 0.23743922204213938)
the data fails to capture his personality or anything meaningful about him.
NOTER_PAGE: (26 0.5347526612398247 . 0.18557536466774716)
the data used by these algorithms is not all there is to know about people.
NOTER_PAGE: (26 0.5792110206637445 . 0.3549432739059967)
B. THE UNFALSIFIABILITY PROBLEM
NOTER_PAGE: (27 0.11396368190356919 . 0.31766612641815234)
Algorithmic predictions can’t be established as true or false until some future date – and sometimes never at all.
NOTER_PAGE: (27 0.15529117094552286 . 0.30551053484602914)
Predictions that are unfalsifiable can readily evade accountability.
NOTER_PAGE: (27 0.3437695679398873 . 0.1944894651539708)
Although the accuracy of predictions that will vest at a certain point can be determined after they vest, decisions are made based on them beforehand. These decisions have consequences at the time of the decision. Waiting to challenge predictions until after they vest will often be too late.
NOTER_PAGE: (27 0.5241077019411396 . 0.2098865478119935)
C. THE PREEMPTIVE INTERVENTION PROBLEM
NOTER_PAGE: (27 0.6468378209142142 . 0.26580226904376014)
interventions make it even more difficult to assess the accuracy of a prediction.
NOTER_PAGE: (27 0.7006887914840325 . 0.306320907617504)
Was the prediction wrong? Or was the prediction correct and the training program effectively reduced the risk? The company’s intervention makes it difficult to evaluate the accuracy of the prediction. Proponents of the algorithm will proclaim that the algorithm prevented accidents – a compelling narrative that can be hard to refute.
NOTER_PAGE: (28 0.21540388227927362 . 0.3784440842787682)
preemptive intervention prevents the prediction from ever vesting, and it can lead to the false narrative that the prediction was correct
NOTER_PAGE: (28 0.35817157169693176 . 0.552674230145867)
there is little the driver can do to challenge the algorithm as it relates to him.
NOTER_PAGE: (28 0.7226048841577958 . 0.6961102106969206)
Imagine if the company failed to respond to the algorithm’s predictions, and Driver X had a crash resulting in the death of many people. Litigation would surely ensue,
NOTER_PAGE: (29 0.21540388227927362 . 0.32009724473257695)
D. THE SELF-FULFILLING PROPHECY PROBLEM
NOTER_PAGE: (29 0.39386349405134624 . 0.24473257698541329)
NOTER_PAGE: (29 0.5203506574827802 . 0.20016207455429497)
NOTER_PAGE: (29 0.5992485911083281 . 0.6102106969205834)
the student, upon learning about her “probable” future, decides that continuing on at the university is pointless
NOTER_PAGE: (30 0.18659987476518472 . 0.4165316045380875)
individuals aren’t given the chance to prove the predictions wrong.
NOTER_PAGE: (30 0.5140889167188478 . 0.19611021069692058)
insufficient data is gathered about instances when algorithms make mistakes.
NOTER_PAGE: (31 0.3149655604257984 . 0.5259319286871961)
E. RACING TOWARDS THE PREDICTION SOCIETY
NOTER_PAGE: (31 0.4032561051972448 . 0.25364667747163694)
1. Creating the Future
NOTER_PAGE: (31 0.5504070131496556 . 0.19935170178282008)
The prediction emerges from the users of these algorithms who assume that correlations in the past will repeat in the future. The algorithms themselves are thus not actually predicting; it is the users of the algorithms who are making predictions based on the strength of this underlying assumption.
NOTER_PAGE: (32 0.1377582968065122 . 0.30307941653160453)
Algorithmic predictions not only forecast the future; they also create it.
NOTER_PAGE: (32 0.24295554164057606 . 0.5040518638573743)
Because predictions are virtually impossible to falsify, individuals have little recourse. They can either decry the predictions and suffer the consequences. Or, they can try to play the game and do actions that might influence the algorithms.153 Instead of challenging the predictions, they might focus on trying to achieve better scores. In this way, predictions can become tyrannous; they can force people to play along.
NOTER_PAGE: (32 0.4370695053224796 . 0.20178282009724471)
NOTER_PAGE: (32 0.5986224170319349 . 0.6004862236628848)
2. The Powerlessness of the Predicted
NOTER_PAGE: (33 0.5140889167188478 . 0.1871961102106969)
as predictions are increasingly used, people become governed by predictions.
NOTER_PAGE: (33 0.6462116468378208 . 0.31604538087520256)
For many entities, their primary goal is not accuracy – as long as the predictions are somewhat better than chance, they will suffice. Nor is their primary goal to make the world a better place. In many cases, the aim is efficiency – saving time and money.
NOTER_PAGE: (33 0.7244834063869755 . 0.6045380875202593)
Organizations might rejoice in the 80% accuracy rate and write off the 10% false positives as a small cost. They might justify such action on the fact that not relying on the algorithm will yield even worse results, as human predictions might be significantly more inaccurate.
NOTER_PAGE: (34 0.3919849718221665 . 0.41329011345218797)
If the algorithm can be used to make a prediction that appears to be more accurate than ordinary human judgment, why not use it?
NOTER_PAGE: (34 0.5053224796493425 . 0.20178282009724471)
In practice, although the algorithm is right only 80% of the time, it will likely be relied upon 100% of the time.
NOTER_PAGE: (34 0.7050720100187852 . 0.36709886547811993)
decisionmakers can readily mistake an algorithm’s stated probability with its accuracy. An algorithm’s creators might claim a high rate of probability, making people trust the algorithm more. But just because an algorithm’s predictions are claimed to be at a high likelihood doesn’t mean that the claim is correct.
NOTER_PAGE: (35 0.18346900438321853 . 0.2941653160453809)
NOTER_PAGE: (35 0.4414527238572323 . 0.32009724473257695)
3. A Future of Predictions
NOTER_PAGE: (35 0.48278021289918593 . 0.2034035656401945)
“current practices mark a shift from quantification of social statistics in order to describe and predict relationships to quantification of social relationships in order to monitor and control them.”
NOTER_PAGE: (36 0.23544145272385722 . 0.30064829821717987)
humans have always made predictions. But human predictions are not systematic. Algorithmic predictions are different
NOTER_PAGE: (36 0.31308703819661865 . 0.2917341977309562)
punishing people for crimes they haven’t yet committed crosses an ethical line and is fundamentally at odds with basic concepts of fairness.
NOTER_PAGE: (36 0.5685660613650595 . 0.34927066450567257)
Algorithms, however, make these situations more ethically troubling because they are more mechanical, systematic, and consistent than human predictions.
NOTER_PAGE: (36 0.736380713838447 . 0.2560777957860616)
NOTER_PAGE: (37 0.3625547902316844 . 0.47487844408427876)
Instead of a grand Borgesian library with an infinitude of tales, the imaginatively-stultified Predicticon allows only the same predictable stories.
NOTER_PAGE: (37 0.40701314965560426 . 0.3824959481361426)
“algorithmic prediction effectively punishes the underlying individual for membership of a statistical group.”
NOTER_PAGE: (37 0.7852222917971196 . 0.293354943273906)
Reference Class Problem
III. ALGORITHMIC PREDICTIONS AND THE LAW
NOTER_PAGE: (38 0.45021916092673764 . 0.25688816855753643)
law has failed to single out predictions and treat them differently from other inferences.
NOTER_PAGE: (38 0.5479023168440826 . 0.41815235008103724)
A. LACK OF A TEMPORAL DIMENSION
NOTER_PAGE: (38 0.675641828428303 . 0.31685575364667745)
GDPR’s definition turns on automation and profiling, and it treats inferences of the past, present, and future the same.
NOTER_PAGE: (39 0.24796493425172197 . 0.3241491085899514)
NOTER_PAGE: (40 0.20037570444583594 . 0.49270664505672607)
GDPR focuses too myopically on automation as the problem.
NOTER_PAGE: (40 0.4082654978083907 . 0.38411669367909235)
adding a “human in the loop” does not cleanse away problematic decisions and can make them worse.
NOTER_PAGE: (40 0.4702567313713212 . 0.33306320907617504)
studies demonstrate that people are overly deferential to automated systems, ignore errors in such systems, and override algorithmic decisions at the wrong times.
NOTER_PAGE: (40 0.6142767689417657 . 0.46029173419773095)
Automation Bias
predictions can, in essence, be human-washed.
NOTER_PAGE: (41 0.23293675641828426 . 0.32009724473257695)
For human involvement to be the answer, the law must set forth exactly how humans would ameliorate the problems with algorithmic predictions in particular cases.
NOTER_PAGE: (41 0.31308703819661865 . 0.40275526742301454)
Even if a human were to override the algorithm and make a prediction based on the human’s experience or hunch, the prediction would still be unfalsifiable.
NOTER_PAGE: (42 0.4859110832811521 . 0.6004862236628848)
The problems we discussed emerge from the practice of prediction; algorithms exacerbate these problems through their automation. Tempering algorithms with humans merely focuses on the automation dimension, but the problems with prediction still remain. Ultimately, the law should better address predictions about people, whether algorithmic or human or hybrid.
NOTER_PAGE: (42 0.6480901690670006 . 0.20016207455429497)
B. THE TRUE-FALSE DICHOTOMY
NOTER_PAGE: (42 0.7413901064495929 . 0.3387358184764992)
privacy law struggles to handle anything that falls in between the binary poles of truth and falsity.
NOTER_PAGE: (43 0.10895428929242329 . 0.5421393841166936)
a matter asserted in a prediction is often not true or false.
NOTER_PAGE: (43 0.1696931747025673 . 0.3452188006482982)
To prevail under these torts, plaintiffs must prove falsity, but a prediction isn’t false. A prediction is, to some extent, an opinion
NOTER_PAGE: (43 0.40763932373199746 . 0.32820097244732577)
C. THE FAILURE OF RECTIFICATION RIGHTS
NOTER_PAGE: (43 0.5929868503443957 . 0.2771474878444084)
Unverifiable predictions are not inaccurate. The real issue is whether predictions are just, fair, and not causing unwarranted harm.
NOTER_PAGE: (44 0.29805886036318097 . 0.506482982171799)
According to the Guidelines, in prediction cases, however, false inferences are considered not necessarily inaccurate if they are statistically “correct.”
NOTER_PAGE: (45 0.1008140262993112 . 0.7212317666126418)
The soundness of the underlying data and method doesn’t guarantee that the prediction is accurate.
NOTER_PAGE: (45 0.4345648090169067 . 0.5413290113452188)
The FCRA focuses far too heavily on the truth-falsity dichotomy and has scant protections against unfair and harmful predictions.
NOTER_PAGE: (46 0.45021916092673764 . 0.19935170178282008)
D. THE LIMITATIONS OF INDIVIDUAL PRIVACY RIGHTS
NOTER_PAGE: (46 0.6743894802755166 . 0.2252836304700162)
Many privacy and data protection laws provide individuals with rights to transparency as well as rights to object, contest, or opt out.213 These rights often fail to adequately address prediction problems.
NOTER_PAGE: (46 0.7094552285535378 . 0.20907617504051862)
Rights place the burden on individuals to manage their own privacy, a task individuals are ill-equipped to do.
NOTER_PAGE: (47 0.1859737006887915 . 0.2917341977309562)
nearly impossible for individuals to understand the particular risks
NOTER_PAGE: (47 0.21540388227927362 . 0.30307941653160453)
individuals would have to monitor each algorithm constantly.
NOTER_PAGE: (47 0.3963681903569192 . 0.29821717990275526)
regulation of algorithmic predictions will fail if merely aimed at specific individuals.
NOTER_PAGE: (47 0.5510331872260488 . 0.326580226904376)
E. THE CHALLENGES OF ANTI-DISCRIMINATION LAW
NOTER_PAGE: (47 0.6399499060738885 . 0.2325769854132901)
prediction problems are not necessarily anti-discrimination problems.
NOTER_PAGE: (47 0.7288666249217282 . 0.2025931928687196)
Algorithmic predictions can conceal discrimination by relying on proxy data. This might not be intentional.
NOTER_PAGE: (49 0.24546023794614902 . 0.20097244732576985)
algorithmic predictions can create new categories of discrimination
NOTER_PAGE: (49 0.4690043832185347 . 0.19773095623987033)
The algorithm might be picking up on existing bias against short people, but because height is not a protected category in anti- discrimination law, the law will do little to stop the use of height in the equation. The result might be an increase in height discrimination.
NOTER_PAGE: (49 0.519724483406387 . 0.3184764991896272)
IV. REGULATING ALGORITHMIC PREDICTIONS
NOTER_PAGE: (50 0.4633688165309956 . 0.2771474878444084)
A. RECOGNIZING ALGORITHMIC PREDICTIONS
NOTER_PAGE: (50 0.662492172824045 . 0.26742301458670986)
Algorithmic predictions should be defined under existing data protection or privacy laws. These laws can lay down additional rules
NOTER_PAGE: (50 0.773324984345648 . 0.20583468395461912)
B. ADDRESSING PREDICTION PROBLEMS
NOTER_PAGE: (51 0.1577958672510958 . 0.2957860615883306)
algorithmic predictions are challenging to regulate because they demand both a scientific approach and a humanities approach.
NOTER_PAGE: (51 0.1897307451471509 . 0.39222042139384117)
1. A Scientific Approach
NOTER_PAGE: (51 0.3287413901064496 . 0.19773095623987033)
law should demand scientific rigor for algorithmic predictions.
NOTER_PAGE: (51 0.4145272385723231 . 0.5834683954619124)
machine learning algorithms are quite difficult to explain to individuals, who are often ill-equipped to comprehend them.
NOTER_PAGE: (54 0.1728240450845335 . 0.2025931928687196)
NOTER_PAGE: (54 0.5009392611145899 . 0.7495948136142625)
NOTER_PAGE: (54 0.5691922354414527 . 0.2034035656401945)
2. A Humanities Approach
NOTER_PAGE: (55 0.4082654978083907 . 0.20178282009724471)
broad standards such as “unfairness” can be useful because of the multifarious and evolving issues involved with algorithmic predictions.
NOTER_PAGE: (56 0.22792736380713838 . 0.486223662884927)
competing conceptions of fairness.
NOTER_PAGE: (57 0.184721352536005 . 0.4189627228525121)
C. REGULATORY CHALLENGES
NOTER_PAGE: (58 0.18534752661239823 . 0.3411669367909238)
Regulation should avoid trigger points such as algorithms or automation, as this could exclude some problematic predictions. Instead, the law should directly confront the problems we discussed.
NOTER_PAGE: (58 0.5072010018785222 . 0.20421393841166935)
The law should consider the following questions: What the prediction is used for? Who is the beneficiary of the predictive system? Who is making the prediction? A government entity? A private company? A lone individual? For what purpose?
NOTER_PAGE: (58 0.6800250469630557 . 0.20421393841166935)
There are some situations where the law should completely bar the use of algorithmic predictions – even when highly accurate.
NOTER_PAGE: (58 0.754539762053851 . 0.20421393841166935)
When a prediction is unverifiable, there must be a possibility of escape from its effects.
NOTER_PAGE: (60 0.15216030056355667 . 0.4246353322528363)
There must be a reasonable point at which a prediction expires.
NOTER_PAGE: (60 0.29618033813400124 . 0.5810372771474878)
chance has virtues.
NOTER_PAGE: (61 0.5003130870381965 . 0.5567260940032415)
Far from lamenting our lack of predictive powers, we should accept the surprises and the new challenges and opportunities they bring.
NOTER_PAGE: (61 0.754539762053851 . 0.4278768233387358)
Antidiscrimination forces nitpicked the opposition’s statistics, but mostly conceded the past to insurers. Yet they kept winning because they made a better case for the future, a case that statistics could not touch.”
NOTER_PAGE: (62 0.5353788353162179 . 0.5348460291734197)
laws viewed preventing fossilization as more important than individualizing rates
NOTER_PAGE: (62 0.7201001878522229 . 0.4497568881685575)
The laws strived to base insurance decisions on how the world should and can be not on how it is.
NOTER_PAGE: (63 0.10644959298685033 . 0.5081037277147488)
CONCLUSION
NOTER_PAGE: (64 0.17908578584846588 . 0.4051863857374392)
predictions shift control over people’s future, taking it away from individuals and giving the power to entities to dictate what people’s future will be. It is essential that we turn our focus to algorithmic predictions and regulate them rigorously. Our future depends on it.
NOTER_PAGE: (64 0.43018159048215404 . 0.19935170178282008)