Second, algorithms inevitably make predictions that discriminate against groups that are different from the average. Remember, COMPAS rates defendants based on more than 100 factors such as criminal history, sex and age. It doesn’t take race into consideration. Indeed, when mathematically scrutinised, it appears that the algorithm does not discriminate by race – someone classified as high risk has the same probability of reoffending irrespective of race. So the algorithm’s prediction accuracy does not vary by race, yet, when further scrutinised, the results are imbalanced. A greater share of black defendants (a minority with a higher recidivism rate than average) who do not reoffend are classified as high risk compared to the average. Given the former, the latter is inevitable: there’s a mathematical limit to how fair any algorithm can be.
How so? Sam Corbett-Davies, Emma Pierson, Sharad Goel (all at Stanford) and Avi Feller (at Berkeley), explain further in this article. Black defendants are more likely to be classified as high risk. Yes, race isn’t part of the data used, but factors that predict reoffending vary by race. For example, if previous convictions and propensity to reoffend are correlated, and if more black defendants have previous convictions, then an algorithm that uses previous convictions to calculate risk will rate black defendants as higher risk than white defendants. The point? If a greater proportion of black defendants are classified as high risk, and if high-risk classification is not always perfect, then a greater share of black defendants who do not reoffend will be classified as high risk. And this is despite the algorithm’s prediction accuracy being independent of race. This is far from fair, and it’s mathematically inevitable.
Four lessons for people (not machines)
It’s true that algorithms can generate biased outcomes that, if left unchecked, can amplify over time. But they do the job they’ve been designed to do. If we crave objectivity and consistency, let’s put the onus back on people to improve the design, use and audit of algorithms.
1. Use algorithms as one part of the decision-making process
Stanislav Petrov, who passed away last May, was a lieutenant colonel in the Soviet Union's Air Defense Forces, and to some he’s known as the man who saved the world. On 26 September 1983, an algorithm using satellite data told Petrov that the US had launched five nuclear-armed intercontinental ballistic missiles. In one interview, he said, “The siren howled, but I just sat there for a few seconds, staring at the big, back-lit, red screen with the word 'launch' on it.” His gut feeling at the time was that something wasn’t right; if he was to expect all-out nuclear war, why were so few missiles being launched? He went against protocol and didn’t press the “button”. It turned out that there had been a computer malfunction and by not escalating the alarm he avoided a disastrous nuclear war.
The missile-detection system was surely helpful to the Soviet (and NATO) military. But as Petrov helped design it, he knew the limitations. With great consideration, he decided to rely on his own judgement more than the information gleaned from an intelligent computer.
Beyond missile defence systems, it’s important to recognise that algorithms can make mistakes. Whenever we design decision tools that use algorithms, we have the responsibility to create processes that empower people to still apply their judgement and common sense.
2. Educate users on the limitations of algorithms
As Petrov’s example shows, algorithms are not a silver bullet (or silver missile for that matter). Calculations are based on the average; they offer probability, not certainty. While Petrov understood this, it’s not clear that most algorithm users – programmers, business leaders, policymakers – do. Certainly, more can be done to educate people on the responsible and effective use of algorithms through formal education and on-the-job training.
Moreover, algorithmic recommendations must be presented in such a way that makes the limitations lucid. For example, one of COMPAS’ inadequacies is that it rates defendants on a 1–10 scale. If instead the results were presented as a percentage it would reflect probability, chances and odds rather than a fait accompli. What’s more, being scored 10 out of 10 suggests that a person is 10 times more likely to reoffend than someone receiving the best possible score of 1 out of 10. This is not the case. There is a greater probability, but it’s actually only 3.8 times higher.
3. Prepare for the audit of algorithms
In the way that manufacturing and service firms expect auditors to come knocking, so too should firms producing and deploying algorithms. In 2015 in the US, the Federal Trade Commission, which protects American consumers, created the Office of Technology Research and Investigation. The initiative was launched to help ensure that customers enjoy the benefits of technological progress without suffering from the risk of unfair practices. As a trusted information source, the office conducts independent studies, evaluates new marketing practices and provides guidance to consumers, businesses and policymakers.
4. Protect your customers and employees by offering the right of human review
A new, stricter European data act is coming into force in May 2018: the General Data Protection Regulation (GDPR). It will build on the existing data protection regulation act and force firms to better handle data. One job of the GDPR is to safeguard people against the risk that a potentially damaging decision is taken without human intervention. So, for instance, if you’re rejected for a loan because your online application was declined automatically by an algorithm, you have the right for human review. The credit provider must then go through manual underwriting processes and check the original decision.
This signals a seismic shift and it’s a topic receiving attention worldwide, such as from German chancellor Angela Merkel who calls for “less secrecy” and more “informed citizens”.
The bigger picture
Let’s rewind to Loomis, who pleaded guilty to fleeing from an officer in 2013. After Loomis appealed his sentence, the case was referred to the Wisconsin Supreme Court. The attorney general’s office defended the use of COMPAS. In July 2016 the Wisconsin Supreme Court ruled that the algorithm can be used as one component of the decision-making process to aid sentencing decisions.
The court challenged the judges to understand COMPAS’s limitations to prevent misuse of the system; knowing what your algorithm can tell you is as important as knowing what it can’t. Petrov, whose actions avoided World War III, taught us the same: despite the algorithm’s big flashing press-me button, he didn’t. Petrov made the ultimate judgement call. The challenge for everyone affected by algorithms now is to ensure that we push the debate – the lessons, the limitations, the opportunities – on algorithmic ethics forward.