Accountability in a computerized society

Nissenbaum advocates strict liability for software makers.

cf Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning

Notes

The extensive presence of computing in these many spheres of life suggests two related concerns. The one is a concern with achieving a suitable degree of reliability and safety for these systems so as to minimize risks and harms; the other is a concern with entrenching and maintaining in those sectors of society that produce and purvey computing technologies a robust culture of accountability, or answerability, for their impacts.

NOTER_PAGE: (1 0.752435064935065 . 0.1691176470588235)

urges attention to the neglected status of accountability for the impacts of computing, specifically for the harms and risks of faulty and malfunctioning systems.

NOTER_PAGE: (2 0.23295454545454547 . 0.19747899159663865)

Accountability, Blame, Responsibility – Conceptual framework

NOTER_PAGE: (3 0.18425324675324675 . 0.15231092436974789)

The Problem of Many Hands

NOTER_PAGE: (4 0.6176948051948052 . 0.15021008403361344)

Responsibility, characteristically understood and traditionally analyzed in terms of a single individual, does not easily generalize to collective action.

NOTER_PAGE: (4 0.752435064935065 . 0.5031512605042017)

Where a mishap is the work of “many hands,” it may not be obvious who is to blame because frequently its most salient and immediate causal antecedents do not converge with its locus of decision making.

NOTER_PAGE: (5 0.10551948051948053 . 0.16491596638655462)

This obscuring of accountability can come about in different ways. In some cases, it may be the result of intentional planning, a conscious means applied by the leaders of an organization to avoid responsibility for negative outcomes, or it may be an unintended consequence of a hierarchical management in which individuals with the greatest decision-making powers are only distantly related to the causal outcome of their decisions.

NOTER_PAGE: (5 0.3693181818181818 . 0.6239495798319328)

Computing is particularly vulnerable to the obstacles of many hands.

NOTER_PAGE: (5 0.573051948051948 . 0.1680672268907563)

most software systems in use are produced in institutional settings,

NOTER_PAGE: (5 0.5876623376623377 . 0.21428571428571427)

computer systems themselves, usually not monolithic, are constructed out of segments or modules.

NOTER_PAGE: (5 0.6225649350649352 . 0.6512605042016807)

many systems being developed and already in use operate on top of other systems

NOTER_PAGE: (5 0.75 . 0.7100840336134453)

unforeseen incompatibilities

NOTER_PAGE: (5 0.7978896103896105 . 0.5672268907563025)

depends on the symbiotic relationship of machine with computer system.

NOTER_PAGE: (5 0.8449675324675325 . 0.5766806722689075)

origin of the malfunction was traced not to a single source, but to numerous faults, which included at least two significant software coding errors (“bugs”) and a faulty microswitch. The impact of these faults was exacerbated by the absence of hardware interlocks, obscure error messages, inadequate testing and quality assurance, exaggerated claims about the reliability of the system in AECL’s safety analysis, and in at least two cases, negligence on the parts of the hospitals

NOTER_PAGE: (6 0.34172077922077926 . 0.3172268907563025)

though the conditions of many hands might indeed obscure accountability, they do not imply that answerability can be foregone.

NOTER_PAGE: (6 0.6331168831168832 . 0.3644957983193277)

accountability for the Therac-25 does not rest with the machine technicians because by and large they were not at fault in any way relevant to the harms and because the control they exercised over the machine’s function was restricted to a highly limited spectrum of possibilities.

NOTER_PAGE: (7 0.08603896103896104 . 0.3739495798319328)

clear evidence of inadequate software engineering, testing and risk assessment.

NOTER_PAGE: (7 0.16396103896103897 . 0.24474789915966386)

many hands obscured accountability by diminishing in key individuals a sense of responsibility

NOTER_PAGE: (7 0.8076298701298702 . 0.7132352941176471)

if computer technology is increasingly produced by “many hands,” and if, as seems to be endemic to many hands situations, we lose touch with who is accountable (such as occurred with the Therac-25), then we are apt to discover a disconcerting array of computers in use for which no one is answerable.

NOTER_PAGE: (8 0.1582792207792208 . 0.22478991596638653)

Bugs

NOTER_PAGE: (8 0.2824675324675325 . 0.14915966386554622)

During construction, hundreds of bridge workers succumbed to a mysterious disease then referred to as “the bends,” or “caisson disease.”14 Although the working conditions and inadequate response from medical staff were responsible for the disease, we cannot assign blame for the harms suffered by the workers or find any individual or distinct group, such as the bridge companies, their chief engineers, or even their medical staff, accountable because causes and treatments of the disease were beyond the scope of medical science of the day.

NOTER_PAGE: (9 0.17775974025974028 . 0.21953781512605042)

When we conceive of bugs as an inevitable byproduct of programming we are likely to judge bug-related failures in the way we judged early handling of the bends: inevitable, albeit unfortunate, consequence of a glorious new technology for which we hold no one accountable.

NOTER_PAGE: (9 0.7175324675324676 . 0.20903361344537813)

if experts in the field deny that such a distinction can be drawn, in view of the inevitability of bugs and their potential hazard, it is reasonable to think that the field of computing is not yet ready for the various uses to which it is being put.

NOTER_PAGE: (10 0.1444805194805195 . 0.2310924369747899)

The Computer as Scapegoat

NOTER_PAGE: (10 0.23457792207792208 . 0.14915966386554622)

having found one explanation for an error or injury, the further role and responsibility of human agents tend to be underestimated

NOTER_PAGE: (10 0.3360389610389611 . 0.3403361344537815)

computers perform tasks previously performed by humans in positions of responsibility. They calculate, decide, control, and remember. For this reason, and perhaps even more deeply rooted psychological reasons (Turkle, 1984), people attribute to computers and not to other inanimate objects (like bullets) the array of mental properties, such as intentions, desires, thoughts, preferences, that lead us to judge human action faulty and make humans responsible for their actions.

NOTER_PAGE: (10 0.5868506493506493 . 0.34243697478991597)

a person cites a computer because she is genuinely baffled about who is responsible.

NOTER_PAGE: (11 0.08766233766233766 . 0.4054621848739496)

Ownership without Liability

NOTER_PAGE: (11 0.35064935064935066 . 0.15126050420168066)

absent in these debates is any reference to owners’ responsibilities.

NOTER_PAGE: (11 0.5665584415584416 . 0.4065126050420168)

the trend in the software industry is to demand maximal property protection while denying, to the extent possible, accountability.

NOTER_PAGE: (11 0.7378246753246753 . 0.38655462184873945)

software is released in society, for which users bear the risks, while those who are in the best position to take responsibility for potential harms and risks appear unwilling to do so.

NOTER_PAGE: (12 0.28165584415584416 . 0.3004201680672269)

Restoring Accountability

NOTER_PAGE: (12 0.4707792207792208 . 0.148109243697479)

An Explicit Standard of Care

NOTER_PAGE: (12 0.752435064935065 . 0.15441176470588236)

it is important to have a good sense of whether or when the “best efforts” as recognized by a field – especially one as widely applied as computing – are good enough for the many uses to which they are put.

NOTER_PAGE: (13 0.3725649350649351 . 0.21428571428571427)

Distinguishing Accountability from Liability

NOTER_PAGE: (13 0.4577922077922078 . 0.14915966386554622)

The problem of many hands is profound and seems unlikely to yield easily to a general, or slick, solution.

NOTER_PAGE: (13 0.7970779220779222 . 0.1691176470588235)

if all we care about is liability, it makes sense to share the burden of compensation among the collective in order to lighten the burden of each individual.

NOTER_PAGE: (14 0.12337662337662339 . 0.5546218487394958)

In contrast, where we care about accountability, many hands do not offer a means of lessening or escaping its burden. No matter how many agents there are, each may be held equally and fully answerable for a given harm.

NOTER_PAGE: (14 0.19237012987012989 . 0.3560924369747899)

Strict Liability and Producer Responsibility

NOTER_PAGE: (15 0.2962662337662338 . 0.15126050420168066)

serious consideration be given to a policy of strict liability for computer system failure, in particular for those sold as consumer products in mass markets.

NOTER_PAGE: (15 0.4675324675324676 . 0.22058823529411764)

incentive to sellers of consumer products and owners of potentially hazardous property to take extraordinary care.

NOTER_PAGE: (15 0.7386363636363636 . 0.5199579831932772)

compensation for victims from those best able to afford it,

NOTER_PAGE: (15 0.7686688311688312 . 0.14285714285714285)

reduces the cost of litigation by eliminating the onerous task of proving fault.

NOTER_PAGE: (15 0.7897727272727273 . 0.25630252100840334)

Because of the prohibitive cost of bearing the full risk of malfunction and injury, many an innovation might not be pursued

NOTER_PAGE: (15 0.8449675324675325 . 0.5609243697478992)

Because of the always- lurking possibility of bugs, software seems to be precisely the type of artifact for which strict liability is appropriate;

NOTER_PAGE: (16 0.42694805194805197 . 0.3245798319327731)