Designing, developing, and deploying artificial intelligence systems: Lessons from and for the public sector - part one

Previous Center authors Kevin Desouza and Gregory Dawson and I recently wrote a paper on Artificial Intelligence and the public sector that was published in Business Horizons, a Journal of the Kelley School of Business, Indiana University.  This article will appear on our blog in a three-part series to include background information on AI and cognitive computing; designing, developing, and deploying cognitive computing systems, and harnessing new technologies.  This blog is the first in the three-part series.

1. AI’s history and value capture potential

Research and development of artificial intelligence (AI) systems have a rich history. There was a flurry of interest when it was originally conceptualized in the 1950s, but it quickly fizzled in the face of AI’s technical realities. Advances in information and computational sciences over the last decade have provided the resources necessary to finally capitalize on the early discoveries and technology underpinning AI systems. As noted by Kai-Fu Lee (2018, p. 12), our current innovations in AI are “merely the application of the past decade’s breakthrough - primarily deep learning but also complementary technologies like reinforcement learning and transfer learning - to new problems.” Today, AI applications can be found in organizations across every sector and include chatbots that help customers navigate websites, predictive analytics systems used for fraud detection, augmented decision-support systems for knowledge workers, and semi- and fully-autonomous systems in transportation, defense, and healthcare.

For the last 6 years, we studied AI systems across public, private, and nonprofit sectors. Our projects spanned multiple industry sectors, including healthcare, law enforcement, education, social services, defense, finance, management consulting, and infrastructure engineering. During these engagements, we had a firsthand view of critical issues organizations must contend with as they design, develop, deploy, and assess AI systems.

While most AI projects are complex, we assert that those within the public sector present some unique challenges:

• The public sector must contend with complex policy, societal, legal, and economic elements that might be skirted by their privatesector counterparts;
• Public-sector AI projects must advance the public good (Cath, Wachter, Mittelstadt, Taddeo, & Floridi, 2018) yet also deliver public value (Crawford, 2016a);
• These projects must go beyond simple cost and efficiency gains to satisfy a richer and diverse set of stakeholders who may have conflicting agendas;
• The need for transparency (Bryson & Winfield, 2017; Edwards & Veale, 2017) and fairness (Chouldechova, 2017; Crawford, 2016) in decision making and system operations adds to the complexity of public-sector AI projects; and
• Given that public-sector projects and systems are taxpayer-funded, these efforts face regular scrutiny and oversight that is generally not seen in the private sector (BBC News, 2019).

While the aggregation of these factors is unique to the public sector, individual components apply to virtually every domain. For example, a nonprofit may have a much more diverse group of stakeholders than a privately held business, but a privately held business may have a higher requirement for cost and efficiency considerations. Thus, we believe that an examination of AI in the public sector can provide insights applicable to all organizations.

In this article, we share reflections and insights from our experience with AI projects in the public sector. Within the phases of AI system engineering and implementation, we organized our findings into four thematic domains (1) data, (2) technology, (3) organizational, and (4) environmentaldand examine these domains relative to the phases of AI.

2. Background on cognitive computing systems

AI systems are often considered part of a larger set of cognitive computing systems (CCSs; Desouza, 2018). CCSs, as the name implies, have cognition due to their learning functions. A CCS can learn in one of two basic ways: supervised and unsupervised. In supervised learning, each record in the training dataset is tagged with its correct classification so that the machine learns what makes a record more or less likely to be in each group. In a fraud detection exercise, each record is tagged as either fraudulent or non-fraudulent, and the machine identifies other attributes in the record that help to distinguish the two groups. The system may find that individuals from a certain school (e.g., University of Arizona) are more likely to commit fraud than those individuals from another school (e.g., Arizona State University). The system looks for the best distinguishing attributes to describe one group versus another.

In unsupervised learning, the system discovers previously unknown patterns or groupings in the data. This is common when a state looks to better understand its citizens. In this case, the machine runs, unsupervised, to discover patterns and types of people that exist within the state. A state may discover four different types of citizens along with their prevalence (e.g., few retirees and lots of families with young children). A human interprets the groupings that emerge; using this information, the state can make better-informed decisions about whether to fund more education or more services for seniors. Choosing between supervised and unsupervised learning is an exercise in understanding the trade-offs that exist between the accuracy of the learning and its interpretability. Thus, numerous selection considerations come into play when making the final decision (Lee & Shin, 2020).

Several characteristics differentiate CSSs from other systems. These five characteristics drive development and deployment choices:

1. CSSs learn from both data and human interactions, and both are required for successful deployment;
2. CSSs are context-sensitive and draw on environmental characteristics (e.g., user profiles and previous interactions) to deal with information requests;
3. CSSs recall history (i.e., previous interactions) in developing recommendations and groupings;
4. CSSs interact with humans through natural language processing; and
5. CSSs provide confidence-weighted recommendations (i.e., outcomes) that can be acted upon by humans.

Numerous examples of successful CCS deployments exist in the public sector (Desouza, 2018; Mehr, 2017). The U.S. government has implemented an AI-based chatbot app that helps potential refugees to the U.S. answer a series of questions to help determine required forms as well as assess whether a refugee is eligible for protection. North Carolina uses AI-based chatbots to answer basic help center questions, which makes up approximately 90% of its phone calls. The Mexican government is piloting an AI tool to classify citizen petitions and then route them to the correct department. Finally, the organizers of the Pyeongchang Winter Olympics developed and deployed an AI-based tool for real-time translation.

However, there have been some notable AI failures. Police departments have purchased an AI- based tool that executes real-time facial recognition. Unfortunately, the tool returned a high number of false positives and falsely matched 28 members of Congress with mugshots of unrelated individuals (Santamicone, 2019). The City of Chicago, in an effort to identify people most likely to be involved in a shooting, developed an algorithm to identify such individuals and stop them from making a firearm purchase. Yet, a report by the RAND Corporation showed that the output from the tool is less effective than a most-wanted list and, even worse, targets innocent citizens for police attention (Stroud, 2016).

These examples show the power of CCS deployment in the public sector as well as the potential risks. It is not surprising that technology experts generally agree on the power of AI to transform the economy and society, but remain sharply divided over whether the transformation will be helpful or harmful (Kietzmann & Pitt, 2020). Managers are responsible for deciding if, where, why, and how AI should be adopted so as to capture its benefits while mitigating its risks (Kietzmann & Pitt, 2020).