Image by Gerd Altmann from Pixabay
|
Journal of Science Policy & Governance | Volume 16, Issue 02 | May 27, 2020
|
Policy Memo: Preventing Racial Bias in Federal AI
Morgan Livingston
University of California Berkeley, Interdisciplinary Studies, Berkeley, California 94720 |
Keywords: AI; artificial intelligence; racial bias; diversity; impact assessment; contestability
Executive Summary: Artificial Intelligence (AI) systems are increasingly used by the US federal government to replace or support decision making. AI is a computer-based system trained to recognize patterns in data and to apply these patterns to form predictions about new data for a specific task. AI is often viewed as a neutral technological tool, bringing efficiency, objectivity and accuracy to administrative functions, citizen access to services, and regulatory enforcement. However, AI can also encode and amplify the biases of society. Choices on design, implementation, and use can embed existing racial inequalities into AI, leading to a racially biased AI system producing inaccurate predictions or to harmful consequences for racial groups. Racially discriminatory AI systems have already affected public systems such as criminal justice, healthcare, financial systems and housing. This memo addresses the primary causes for the development, deployment and use of racially biased AI systems and suggests three responses to ensure that federal agencies realize the benefits of AI and protect against racially disparate impact. There are three actions that federal agencies must take to prevent racial bias: 1) increase racial diversity in AI designers, 2) implement AI impact assessment, 3) establish procedures for staff to contest automated decisions. Each proposal addresses a different stage in the lifecycle of AI used by federal agencies and helps align US policy with the Organization for Economic Co-operation and Development (OECD) Principles on Artificial Intelligence.
I. What is racial bias in AI and why is it a problem?
Federal agencies are increasingly adopting Artificial Intelligence (AI) and delegating critical decisions to the technology. Out of the 142 largest federal agencies, 45% use or have considered using AI, for tasks ranging from setting bail to detecting fraud (Engstrom et al. 2020). Although AI can bring efficiency and objectivity to services, AI systems can also magnify systemic inequities. AI can replicate and amplify existing biases, producing predictions with harmful outcomes for a racial group. The causes for bias are both technical and social: the code can be embedded through the biases of the designers and data, and the use of AI can exacerbate bias already existing in a social system.
When used by a federal agency, AI predictions take on power as the basis for critical decisions, or in the case of automated decisions, the cause for immediate impact. The lifecycle of an AI system is an iterative process of defining the problem AI addresses, deciding to use AI, designing, coding, testing, deploying, maintaining and retiring the AI. The impact of an AI system depends on choices made at different stages in the AI lifecycle, including:
Without sufficient safeguards, human choices can incorporate racial bias into AI systems, causing significant impact. Studies show racial bias in AI has already caused harm in many sectors including facial recognition, criminal sentencing, healthcare, and financial services.
The examples above are a selection of the known cases where biases in the design or use of AI led to racially disparate harm.
When used by a federal agency, AI predictions take on power as the basis for critical decisions, or in the case of automated decisions, the cause for immediate impact. The lifecycle of an AI system is an iterative process of defining the problem AI addresses, deciding to use AI, designing, coding, testing, deploying, maintaining and retiring the AI. The impact of an AI system depends on choices made at different stages in the AI lifecycle, including:
- Designers train an AI model to form predictions based on patterns learned in historical data, choosing the dataset the model will learn from, the accuracy of the model’s prediction for different groups, and the testing procedure to evaluate the model.
- Staff deploys the AI system for their use case, choosing whether the AI model is appropriate for their task, how to use the AI predictions and who will manage the AI.
- Users act on the AI predictions, choosing how to manage the AI system and translate the machine output into conclusions with real impact.
Without sufficient safeguards, human choices can incorporate racial bias into AI systems, causing significant impact. Studies show racial bias in AI has already caused harm in many sectors including facial recognition, criminal sentencing, healthcare, and financial services.
- Facial Recognition: Facial recognition tools produce significantly higher false positive rates for African and East Asian individuals than for white individuals (Grother et al. 2019). One commercial tool had a 0.8% error rate for light-skinned males, but 34.7% error rate for dark skinned-females (Buolamwini & Gebru 2018). Disparate errors can lead to law enforcement falsely matching suspects with criminal databases (Snow 2018).
- Criminal Sentencing: COMPAS, an automated risk assessment tool used for criminal sentencing in Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, incorrectly labeled black defendants as future criminals at close to twice the rate as white defendants (Angwin et al. 2019).
- Healthcare: A healthcare algorithm responsible for 200 million people systemically prevented almost 30% of eligible black patients from receiving additional care by giving lower risk scores to black patients than white patients with equal diagnoses (Obermeyer et al. 2019).
- Loans: FinTech firms charged Latinx and African-American loan borrowers 7.9 and 3.6 basis points, respectively, more than equivalent White borrowers, costing a yearly extra $765 million in interest (Bartlett et al. 2019).
The examples above are a selection of the known cases where biases in the design or use of AI led to racially disparate harm.
-Read the full article through download.- |
References
- Almada, Marco. 2019. “Human Intervention in Automated Decision-Making: Toward the Construction of Contestable Systems.” In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law - ICAIL ’19, 2–11. Montreal, QC, Canada: ACM Press. https://doi.org/10.1145/3322640.3326699
- American Civil Liberties Union, et al. 2016. “Predictive Policing Today: A Shared Statement of Civil Rights Concerns.” American Civil Liberties Union. August 31, 2016. https://www.aclu.org/other/statement-concern-about-predictive-policing-aclu-and-16-civil-rights-privacy-racial-justice
- Angwin, Julia, Jeff Larson, Surya Mattu and Lauren Kirchner. 2016. “Machine Bias.” ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
- Barocas, Solon, and Andrew D. Selbst. 2016. “Big Data’s Disparate Impact.” SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2477899
- Bartlett, Robert, Adair Morse, Richard Stanton, and Nancy Wallace. 2019. “Consumer-Lending Discrimination in the FinTech Era.” w25943. Cambridge, MA: National Bureau of Economic Research. https://doi.org/10.3386/w25943
- Buolamwini, Joy, and Timnit Gebru. 2018. "Gender shades: Intersectional accuracy disparities in commercial gender classification." In Conference on fairness, accountability and transparency, pp. 77-91.
- Calo, Ryan. 2017. "Artificial Intelligence Policy: A Primer and Roadmap," U.C. Davis Law Review 51, no. 2: 399-436. https://doi.org/10.2139/ssrn.3015350
- Citron, Danielle Keats. 2007. "Technological due process." Wash. UL Rev. 85: 1249.
- Crawford, Kate, and Ryan Calo. 2016. “There Is a Blind Spot in AI Research.” Nature 538 (7625): 311–13. https://doi.org/10.1038/538311a
- Dwork, Cynthia, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. “Fairness through Awareness.” In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference on - ITCS ’12, 214–26. Cambridge, Massachusetts: ACM Press. https://doi.org/10.1145/2090236.2090255
- Engstrom, David, Daniel Ho, Catherine Sharkey, and Mariano-Florentino Cuellar. 2020. “Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies.” Stanford Law School. https://www-cdn.law.stanford.edu/wp-content/uploads/2020/02/ACUS-AI-Report.pdf http://dx.doi.org/10.2139/ssrn.3551505
- European Commission. 2020. “White Paper on Artificial Intelligence – A European approach to excellence and trust.” https://ec.europa.eu/info/sites/info/files/commission-white-paper-artificial-intelligence-feb2020_en.pdf
- G20 Trade Ministers and Digital Economy Ministers. 2019. “G20 Ministerial Statement on Trade and Digital Economy.” https://www.mofa.go.jp/files/000486596.pdf
- Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. “Datasheets for Datasets.” https://arxiv.org/abs/1803.09010v6
- Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). “Automation bias: a systematic review of frequency, effect mediators, and mitigators.” Journal of the American Medical Informatics Association: JAMIA, 19(1), 121–127. https://doi.org/10.1136/amiajnl-2011-000089
- Google. 2018.“Google Diversity Annual Report 2018.” https://static.googleusercontent.com/media/diversity.google/en//static/pdf/Google_Diversity_annual_report_2018.pdf
- Google. 2020. “What-If Tool.” Google People + AI Research initiative (PAIR). Accessed March 4, 2020. https://pair-code.github.io/what-if-tool/ai-fairness.html
- Grother, Patrick, Mei Ngan, and Kayee Hanaoka. 2019. “Face Recognition Vendor Test Part 3:: Demographic Effects.” NIST IR 8280. Gaithersburg, MD: National Institute of Standards and Technology. https://doi.org/10.6028/NIST.IR.8280
- Hagendorff, Thilo. 2020. “The Ethics of AI Ethics: An Evaluation of Guidelines.” Minds and Machines. https://doi.org/10.1007/s11023-020-09517-8
- Hirsch, Tad et al. 2017. “Designing Contestability: Interaction Design, Machine Learning, and Mental Health.” DIS. Designing Interactive Systems (Conference) vol. 2017: 95-99. https://doi.org/10.1145/3064663.3064703
- IBM. 2020. “AI Fairness 360 Open Source Toolkit.” IBM Research Trusted AI. Accessed March 4, 2020. https://aif360.mybluemix.net/
- Kim, Pauline. 2017. "Auditing Algorithms for Discrimination." University of Pennsylvania Law Review Online 166: 189-204.
- Microsoft. 2019. “Diversity and Inclusion Report 2019.” https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE4aqv1
- Mittelstadt, Brent, Chris Russell, and Sandra Wachter. 2019. “Explaining Explanations in AI.” In Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* ’19, 279–88. Atlanta, GA, USA: ACM Press. https://doi.org/10.1145/3287560.3287574
- Narayanan, Arvind. 2018. “Tutorial: 21 definitions of fairness and their politics.” Conference on Fairness, Accountability, and Transparency, NYC, Feb 23, 2018. https://www.youtube.com/watch?v=jIXIuYdnyyk
- National Institute of Standards and Technology (NIST). 2019. “U.S. Leadership in AI: A Plan for Federal Engagement in Developing Technical Standards and Related Tools.” https://www.nist.gov/system/files/documents/2019/08/10/ai_standards_fedengagement_plan_9aug2019.pdf
- National Science Board. 2019. “Science and Engineering Labor Force.” National Center for Science and Engineering Statistics. September 26, 2019. https://ncses.nsf.gov/pubs/nsb20198/assets/nsb20198.pdf
- Newman, Jessica. 2019. “Toward AI Security: Global Aspirations for a More Resilient Future.” Center for Long-Term Cybersecurity. https://cltc.berkeley.edu/wp-content/uploads/2019/02/CLTC_Cussins_Toward_AI_Security.pdf
- Obermeyer, Ziad, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations.” Science 366(6464): 447–53. https://doi.org/10.1126/science.aax2342
- Organisation for Economic Co-operation and Development (OECD). 2019. “Recommendation of the Council on Artificial Intelligence.” Legal Instrument 0449. https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449
- Reisman, Dillon, Jason Schultz, Kate Crawford, and Meredith Whittaker. 2018. “Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability.” AI Now. https://ainowinstitute.org/aiareport2018.pdf
- Select Committee on Artificial Intelligence. 2019. “The National Artificial Intelligence Research and Development Strategic Plan: 2019 Update,” National Science and Technology Council. https://www.nitrd.gov/pubs/National-AI-RD-Strategy-2019.pdf
- Snow, Jacob. 2018. “Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots.” American Civil Liberties Union. https://www.aclu.org/blog/privacy-technology/surveillance-technologies/amazons-face-recognition-falsely-matched-28
- State v. Loomis, 881 N.W.2d 749, 765 (Wis. 2016).
- Treasury Board of Canada. 2019. “Directive on Automated Decision-Making.” https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=32592
- U.S. Executive Office of the President. 2016. “Preparing for the Future of Artificial Intelligence,” National Science and Technology Council, Washington D.C., October 2016. https://obamawhitehouse.archives.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai.pdf
- U.S. President. Executive Order. “Executive Order 13859 of February 14, 2019, Maintaining American Leadership in Artificial Intelligence.” Code of Federal Regulations, 84(31), p.3967. whitehouse.gov/presidential-actions/executive-order-maintaining-american-leadership-artificial-intelligence/
- Vought, Russel. 2020. “Guidance for the Regulation of Artificial Intelligence Applications.” Executive Office of the President, Office of Management and Budget. January 2020. https://www.whitehouse.gov/wp-content/uploads/2020/01/Draft-OMB-Memo-on-Regulation-of-AI-1-7-19.pdf
- West, S.M., M. Whittaker, and K. Crawford. 2019. “Discriminating Systems: Gender, Race and Power in AI,” AI Now Institute. https://ainowinstitute.org/discriminatingsystems.html
- White House Office of Science and Technology Policy. 2020. “American Artificial Intelligence Initiative: Year One Annual Report.” February, 2020. https://www.whitehouse.gov/wp-content/uploads/2020/02/American-AI-Initiative-One-Year-Annual-Report.pdf
- World Economic Forum. 2019. “Guidelines for AI Procurement.” http://www3.weforum.org/docs/WEF_Guidelines_for_AI_Procurement.pdf
Morgan Livingston is an undergraduate at UC Berkeley studying Technology Policy in the Interdisciplinary Studies Field and Data Science. Morgan is a research assistant at the Berkeley Center for Globalization and Information Technology and focuses on privacy and data law.
DISCLAIMER: The findings and conclusions published herein are solely attributed to the author and not necessarily endorsed or adopted by the Journal of Science Policy and Governance. Articles are distributed in compliance with copyright and trademark agreements.
ISSN 2372-2193
ISSN 2372-2193