Interpretability in AIML System
The Black Box Problem
In early 2020, the Turing Award winner Professor “Geoffrey Hinton” proposed a heatedly debated question about the importance of interpretability in ML systems.
“Suppose you have cancer and you have to choose between a black box AI surgeonthat cannot explain how it works but has a 90% cure rate and a human surgeon with an 80% cure rate. Do you want the AI surgeon to be illegal?”
Is accuracy alone enough to make an AI system trustworthy?
The Black Box Problem
We use complex technologies every day without understanding exactly how they work.
You don’t need to understand the physics of a microwave to heat your food. You simply press a button, and it works.
So why do we expect AI systems to explain themselves?
Get Prince’s stories in your inbox
Join Medium for free to get updates from this writer.
The answer lies in the consequences of their decisions.
If a microwave behaves unexpectedly, you might end up with cold food. But if an AI system rejects your loan application, misdiagnoses a disease, or incorrectly identifies someone as a security threat, the consequences can be far more serious.
Modern machine learning models, particularly deep neural networks, can learn extremely complex patterns from data. However, understanding precisely how they arrive at a particular prediction can be difficult.
This is known as the black box problem.
Interpretability is the extent to which humans can understand how an AI model arrives at its predictions or decisions.
Why Does Interpretability Matter?
First, interpretability is important for users, both business leaders and end users, to understand why a decision is made so that they can trust a model and detect potential biases. (For certain use cases in certain countries, users have a “right to explanation”: a right to be given an explana‐
tion for an output of the algorithm.)
Second, it’s important for developers to be able to debug and improve a model.
Just because interpretability is a requirement doesn’t mean everyone is doing it. As of 2019, only 19% of large companies are working to improve the explainability of their algorithms.
Since most ML research is still evaluated on a single objective, model performance, researchers aren’t incentivized to work on model interpretability. However, interpretability isn’t just optional for most ML use cases in the industry, but a requirement.
source: Chip Huyen’s Book
