In “fintech”, which is where my day job is, many companies create algorithms to calculate people’s ‘risk profiles’. That can mean different things in different contexts. For investment-tech companies, a person’s risk profile usually determines what set of investments to recommend the person. (those investments are then sometimes automated via another algorithm). For credit-tech or lend-tech companies, a risk profile is a measure of how likely the person is to default on their loan, and that determines yes/no decisions, documentation, collateral, lending rate among others.
And speaking of credit, this article by MIT Technology Review describes the creeping influence of black-box algorithms in people’s day to day lives:
“Consumer reporting agencies, including credit bureaus, tenant screening companies, or check verification services, amass this information from a wide range of sources: public records, social media, web browsing, banking activity, app usage, and more. The algorithms then assign people “worthiness” scores, which figure heavily into background checks performed by lenders, employers, landlords, even schools.”
“Their comprehensive influence means that if your score is ruined, it can be nearly impossible to recover. Worse, the algorithms are owned by private companies that don’t divulge how they come to their decisions. Victims can be sent in a downward spiral that sometimes ends in homelessness or a return to their abuser.”
Even if a private company were forced to describe their algorithm, dissecting code in a court case is remarkably different from dissecting facts. As one of the lawyers handling such cases put it, “Am I going to cross-examine an algorithm?”
The unfortunately prescient series Black Mirror aired an episode back in 2016 that described a world ruled by people’s social scores, scores that real estate firms used to screen potential customers. As it turns out, reality has managed to out-dystopia a dystopian fictional series.