The best decisions are not the best advice: making adherence-aware recommendations
The research presents a model for creating "adherence-aware" AI recommendations in expert-in-the-loop decision systems, addressing the challenge of partial human adherence to AI suggestions, and offering solutions that ensure robust performance even when decision-makers only follow AI guidance partially.
The challenge:
While some decisions can be automated, many high-stakes decisions follow an ‘expert-in-loop’ structure. Here, an expert decision-maker (e.g. a doctor) receives information or recommendations from an AI tool and decides on a course of action. Hence, the expert may only partially follow the algorithm’s recommendations. Such partial adherence behavior may lead to catastrophic failure, even when the decision-maker seldom deviates from the algorithm. The question is: How can we develop recommendation systems that are safe in the presence of such partial adherence?
The research presents a model for creating "adherence-aware" AI recommendations in expert-in-the-loop decision systems, addressing the challenge of partial human adherence to AI suggestions, and offering solutions that ensure robust performance even when decision-makers only follow AI guidance partially.
The challenge:
While some decisions can be automated, many high-stakes decisions follow an ‘expert-in-loop’ structure. Here, an expert decision-maker (e.g. a doctor) receives information or recommendations from an AI tool and decides on a course of action. Hence, the expert may only partially follow the algorithm’s recommendations. Such partial adherence behavior may lead to catastrophic failure, even when the decision-maker seldom deviates from the algorithm. The question is: How can we develop recommendation systems that are safe in the presence of such partial adherence?
The research:
We devised a model to account for partial adherence in the context of multi-stage decision-making. On the negative side, we show that deploying an algorithm that ignores the possibility of partial adherence can have a dramatic impact on system performance, and even be worse than having no AI-recommendation at all. On the positive note, our model offers a solution by providing a framework to compute algorithmic recommendations whose performance is resilient in the face of human deviations, hence improving upon both human performance (without any algorithm) and adherence-unaware recommendations. This new framework for “adherence-aware” decision-making provides safe recommendations, with performance guarantees even if the decision-maker only partially follows them.
The impact:
Previous work on human adherence to algorithmic recommendations mostly focused on examining the reasons why humans deviate (e.g., information asymmetry, algorithmic aversion), to understand the impact of partial adherence, and propose solutions to increase human adherence. This research takes a complementary approach, by taking the partial adherence phenomenon as an input and proposing to adjust the algorithmic recommendations instead of the human behaviour. Unlike other approaches, this model is agnostic to the reasons behind partial adherence. Using a framework to enhance AI recommendations by accounting for human deviations promises more robust and effective decision-making in expert-in-the-loop systems.