Imagine an artificial intelligence system that has performed reliably for months. Its recommendations are almost always right, and the employee reviewing them has learned that close scrutiny rarely changes the outcome. Then something unusual happens. The AI makes an error or can’t proceed and needs human intervention. The employee is expected to step in and remains responsible for the outcome.
But does the employee know how to do the task?
I’m a professor of management information systems and the academic director of Drexel University’s Center for Applied AI & Business Analytics. In my research on human-AI relationships, I call this latent risk: the possibility that people may lose capabilities their organizations still depend on them to exercise.
AI can continue to perform well while the people overseeing it become less practiced at exercising independent judgment and less prepared to intervene. The risk becomes latent when the organization still depends on that human capability to step in but has little cause to notice that that capability may be weakening.
The concern is not entirely new. The idea of latent risk builds on decades of research into how people respond to automation. For example, studies of automation bias have shown that people can miss problems or follow automated recommendations even when other evidence suggests they should not.
What is different today is the role AI is beginning to play in organizations. It is beginning to assume larger parts of the analysis and decision-making that people previously performed.
Human in the loop explained.
The success paradox
The better AI becomes, the more the nature of human work changes. Work that once required an employee to analyze a problem and reach a decision may instead be performed by AI, leaving the employee to review its recommendation. The employee is no longer doing the same work with AI assistance—rather, their job has shifted from doing the work to overseeing what the AI does.
This shift is already beginning to change how some companies train employees. Accounting giant Deloitte, for example, is redesigning its graduate audit training as AI takes over some of the manual work traditionally done by junior auditors. The firm is introducing simulations so trainees can still develop the judgment they will need as they move into more complex work.
More than four decades ago, cognitive psychologist Lisanne Bainbridge described one of the “ironies of automation”: As systems take over more of the work, instead of simply having their work loads lightened, people can be left facing new challenges in monitoring the systems and handling the unusual situations the systems cannot. Later studies found another problem: As people become less involved in a task, they can become less prepared to step in when needed. Researchers have called this the out-of-the-loop performance problem.
This reduced involvement may affect more than a person’s readiness to intervene. It can change the capabilities available to the organization.
What work remains for people does not necessarily become easier. It may become harder. These are the situations where experience and independent judgment matter most. The result is a paradox: People may be called upon to exercise independent judgment less often, even as the judgments left to them become more difficult.
The accountability problem
Accountability may not change as quickly as the work does when organizations deploy AI systems. An employee may now oversee far more decisions than they make themselves, while remaining responsible for the outcomes. The organization’s expectations may still reflect how the work was divided before AI took on a larger role. The organization can therefore end up holding people accountable for decisions over which they have less control.
This creates a mismatch. Research on meaningful human control suggests that responsibility should match a person’s authority and ability to act. An employee may retain the authority to override an AI recommendation and remain responsible for the outcome, yet lose the capability to do so. Authority, responsibility, and capability can begin to separate.
Latent risk becomes an organizational design problem, not simply a human performance problem.
The measurement problem
Organizations typically judge AI-enabled processes by measures such as accuracy and productivity. Those numbers may look good even as the human role is changing. Suppose employees override AI recommendations less frequently over time. Is that a sign of growing dependence? Perhaps.
But it could also mean that the AI has become more accurate and there is less reason to intervene. The number of overrides alone tells little about whether employees remain ready to exercise their own judgment.
The key is to understand how the division of work between people and AI is changing. Which decisions have migrated from people to AI? What responsibilities remain with people? Looking at the performance of the employee or the AI separately may miss what is changing between them.
The governance problem
Organizational governance—defining roles and responsibilities—is also under pressure from AI.
Someone may remain formally in the loop with an AI system even as their actual role changes. Defining human and AI roles when a system is introduced is only a starting point. As AI improves and people become accustomed to using it, those roles are likely to change over time. There is nothing inherently wrong with that change.
The problem arises when the relationship changes but the assumptions surrounding it do not. It’s important for organizations to periodically ask: What relationship between people and AI did we design, and what relationship do we actually have today?
In short, organizational governance needs to be dynamic.
Maintaining the human backstop
Not every human capability needs to be preserved. If AI can perform a task well, there is little reason to require people to keep doing the same work. Organizations need to decide which capabilities they still expect people to have when human judgment is required. If people may still be called upon to use a capability, it needs to be maintained.
Overseeing AI is not the same as exercising independent judgment. If people rarely make the decision themselves, the organization cannot assume they will be ready when they need to. One way to test this is to occasionally ask employees to make a decision without first seeing the AI recommendation. The organization needs to know that the human backstop it relies on will actually work when needed.
As AI takes on more work, some human capabilities will inevitably be used less often. Some may no longer matter. Others may still be essential when AI reaches its limits. The challenge is knowing the difference and making that choice deliberately.
For me, the central question is this: Which human capabilities does an organization still depend on, and will they be there when needed?
Murugan Anandarajan is a professor of decision sciences and management information systems at Drexel University.
This article is republished from The Conversation under a Creative Commons license. Read the original article.