Outsourced and automated: how AI companies have taken over government decision-making

Notes

Introduction

NOTER_PAGE: (6 0.03724928366762178 . 0.3818350324374421)

AI systems are different from other products and services traditionally purchased through government procurement: they displace agency decision-making and discretion, often in ways that are difficult to decipher and manage.

NOTER_PAGE: (7 0.22421203438395415 . 0.1936978683966636)

The Risks of Government AI

NOTER_PAGE: (10 0.03796561604584527 . 0.38368860055607046)

“AI system,” is used to describe any system that automates a process, aids human decision-making, or replaces human decision-making—including simpler automated decision-making systems.

NOTER_PAGE: (11 0.8108882521489971 . 0.30954587581093607)

When the outputs of AI systems are flawed or biased, neither agencies nor impacted individuals have the information they need to correct the errors. AI vendors routinely keep this information private using trade secret laws and contractual provisions, even when disclosing information about an AI system’s technical processes and training data could mitigate harm.

NOTER_PAGE: (13 0.1346704871060172 . 0.24930491195551438)

Four common functions of government AI

NOTER_PAGE: (14 0.27363896848137537 . 0.2391102873030584)

Risk Scoring:

NOTER_PAGE: (14 0.3438395415472779 . 0.1927710843373494)

Eligibility Screening:

NOTER_PAGE: (14 0.5050143266475644 . 0.19462465245597776)

Fraud Detection:

NOTER_PAGE: (14 0.6683381088825214 . 0.1890639481000927)

Predictive Policing:

NOTER_PAGE: (14 0.7994269340974212 . 0.1927710843373494)

three levels of complexity in government AI:

NOTER_PAGE: (15 0.15401146131805157 . 0.49953660797034294)

Simple rules-based algorithms,

NOTER_PAGE: (15 0.22277936962750716 . 0.19925857275254866)

Machine-learning algorithms,

NOTER_PAGE: (15 0.32091690544412604 . 0.1936978683966636)

Generative AI systems,

NOTER_PAGE: (15 0.42406876790830944 . 0.1936978683966636)

Privacy Risks: How AI Uses & Abuses Your Data

NOTER_PAGE: (15 0.5952722063037249 . 0.12511584800741427)

To build and maintain many government AI systems, private AI developers rely on commercial databases as well.

NOTER_PAGE: (17 0.4627507163323782 . 0.4040778498609824)

RentGrow, an automated tenant screening service used by the D.C. Housing Authority. As part of its service, RentGrow not only collects information about applicants through cookies and public databases, but also purchases data about applicants’ social media profiles and “intent data”:

NOTER_PAGE: (18 0.10028653295128939 . 0.40593141797961074)

similar to RentGrow, Thomson Reuters pulls data from location services, credit reporting agencies, social media scrapers, public databases, and other data brokers’ databases to train and operate their fraud detection model.

NOTER_PAGE: (18 0.7585959885386819 . 0.11677479147358666)

state agencies need to give Deloitte access to extensive data about benefits applicants—data that Deloitte in turn incorporates within its own commercial database. This data transfer happens without the express consent of applicants, and often exposes sensitive personal data to new cybersecurity vulnerabilities.

NOTER_PAGE: (20 0.6382521489971347 . 0.603336422613531)

Accuracy Risks: When AI Makes Mistakes

NOTER_PAGE: (21 0.10171919770773638 . 0.12696941612604262)

Most companies that provide AI systems to government agencies maintain that the logic of their systems is proprietary,

NOTER_PAGE: (23 0.4613180515759312 . 0.2205746061167748)

in 2022, the agency agreed to transition from MiDAS to Deloitte’s uFACTS

NOTER_PAGE: (25 0.419054441260745 . 0.30954587581093607)

Accountability Risks: Undermining Government

NOTER_PAGE: (25 0.6103151862464183 . 0.12233549582947173)

vendors can keep the procedures their AI systems use to make decisions a secret by claiming that the software and machine-learning models behind their AI systems are “proprietary business information.”

NOTER_PAGE: (25 0.8252148997134671 . 0.5393883225208527)

Widespread AI procurement and protections for trade secrets mean that many government AI systems operate without meaningful public oversight.

NOTER_PAGE: (26 0.2793696275071633 . 0.211306765523633)

Administrative law is not prepared for AI decision- making.

NOTER_PAGE: (26 0.49426934097421205 . 0.12511584800741427)

defended the existence of administrative agencies on the practical assumption that agency expertise and discretion are needed

NOTER_PAGE: (27 0.2793696275071633 . 0.5597775718257646)

agencies’ adoption of AI systems has challenged this assumption.

NOTER_PAGE: (27 0.40186246418338106 . 0.3957367933271548)

When an agency official makes a decision that affects you, there are opportunities for public comment, hearings, or a record supporting the decision. When private vendors make these decisions, agencies and the public are left to rely on procurement procedures and vendor disclosures for accountability. These procedures are not designed to provide substantive oversight over government decision-making, but to promote competition, efficiency, and risk avoidance.

NOTER_PAGE: (28 0.6103151862464183 . 0.4087117701575533)

The AI Vendor Landscape

NOTER_PAGE: (31 0.04083094555873926 . 0.3855421686746988)

Eubanks calls this growing network of government surveillance and AI a “digital poorhouse”: a nationwide web of microphones, cameras, fingerprint scanners, algorithms, and assessments that disproportionately target low- income communities.

NOTER_PAGE: (32 0.7012893982808023 . 0.3030583873957368)

Modern procurement methods are ill-suited to AI systems,

NOTER_PAGE: (33 0.7743553008595988 . 0.18164967562557924)

AI vendors aggressively market their systems to state agencies and state legislatures,

NOTER_PAGE: (34 0.13395415472779368 . 0.4355885078776645)

State agencies struggle to attract employees with AI expertise,

NOTER_PAGE: (34 0.2507163323782235 . 0.17701575532900835)

Bidding Optional: Different Paths to Procurement

NOTER_PAGE: (34 0.44126074498567336 . 0.12048192771084337)

Path 1: Competitive Bidding

NOTER_PAGE: (35 0.25429799426934097 . 0.12511584800741427)

Drafting and submitting competitive bids can take a lot of time and resources,

NOTER_PAGE: (36 0.16189111747851 . 0.35217794253938833)

Path 2: Exemptions from Competitive Bidding

NOTER_PAGE: (36 0.7564469914040114 . 0.12048192771084337)

expedite procurement for routine services, provide flexibility for low-cost contracts, or facilitate government responses to emergencies.

NOTER_PAGE: (37 0.1353868194842407 . 0.3994439295644115)

many states relied on pandemic-era public health exemptions to rapidly procure AI systems to keep government services afloat.

NOTER_PAGE: (37 0.21704871060171918 . 0.3818350324374421)

Deloitte contracted with at least eight states to deploy its unemployment claims management system, uFACTS, making over $410 million without going through a competitive bidding process.

NOTER_PAGE: (37 0.48853868194842404 . 0.5301204819277109)

Path 3: Cooperative Purchasing

NOTER_PAGE: (37 0.7614613180515759 . 0.11770157553290084)

The Major AI Vendors

NOTER_PAGE: (41 0.10458452722063037 . 0.12233549582947173)

the products and services of just ten vendors account for over $715,000,000 in AI contracts across 42 states.

NOTER_PAGE: (41 0.4068767908309455 . 0.2882298424467099)

The Anatomy of an AI Contract

NOTER_PAGE: (45 0.10673352435530085 . 0.12140871177015755)

Because the contract includes a per- search fee schedule instead of a total price, the actual amount of money flowing from D.C. government to RentGrow continually changes

NOTER_PAGE: (49 0.25358166189111747 . 0.17145505097312327)

because RentGrow organizes its tenant screening system as a service it provides instead of a product it furnishes to the purchasing agency, D.C. government never has direct access to the automated processes

NOTER_PAGE: (49 0.37177650429799425 . 0.45041705282669137)

Recommendations & Reforms

NOTER_PAGE: (54 0.03653295128939828 . 0.3855421686746988)

Robust, transparent, and independent audits of AI systems and their outputs

NOTER_PAGE: (55 0.336676217765043 . 0.11862835959221502)

For some AI applications, the risks are so high that prohibitions on AI use are the only path forward.

NOTER_PAGE: (56 0.5472779369627507 . 0.4198331788693235)

requiring contract provisions that protect individuals and their data will go far to prevent harms from government AI systems

NOTER_PAGE: (58 0.6948424068767908 . 0.5440222428174235)

Improving Data Oversight and Control:

NOTER_PAGE: (59 0.10100286532951289 . 0.17794253938832252)

Imposing Transparency or Reporting Requirements:

NOTER_PAGE: (59 0.47564469914040114 . 0.17516218721038)

Incorporating Sunsetting Clauses or Procedures to Transition Ownership to Agencies:

NOTER_PAGE: (60 0.10315186246418338 . 0.17979610750695088)

Requiring Human Review:

NOTER_PAGE: (60 0.41260744985673353 . 0.17979610750695088)

in one ongoing case, K.W. v. Armstrong, attorneys from the ACLU of Idaho convinced a federal district court that disseminating information about how an assessment tool operated was required under the Due Process Clause of the Fourteenth Amendment.

NOTER_PAGE: (61 0.42478510028653294 . 0.432808155699722)

several features of government AI systems make it difficult for private individuals to identify how they were harmed—let alone litigate their injury in court.

NOTER_PAGE: (61 0.8452722063037249 . 0.6969416126042632)

can use existing legal remedies to unfair or illegal agency decision-making, but they need to know how decisions were made to pursue them.

NOTER_PAGE: (62 0.583810888252149 . 0.26135310472659873)

“private right of action,” which would give individuals harmed by government AI systems a statutory basis for litigating their injury.

NOTER_PAGE: (62 0.7593123209169054 . 0.5903614457831325)

centralized, searchable database of AI documentation, processes, and audit results.

NOTER_PAGE: (62 0.8495702005730659 . 0.3985171455050973)

state agencies should view legal aid organizations as collaborators

NOTER_PAGE: (63 0.6131805157593123 . 0.22520852641334568)

many faulty AI systems have cost state agencies millions more in litigation costs than agencies saved by outsourcing and automating their processes.

NOTER_PAGE: (65 0.36962750716332377 . 0.4281742354031511)

Appendix A Our Methods

NOTER_PAGE: (67 0.252865329512894 . 0.11399443929564412)