AI-use could spell discrimination for Danish welfare recipients: Amnesty

Welfare recipients in Denmark are at risk of becoming overly monitored and discriminated as a result of algorithms and artificial intelligence designed to identify fraud, Amnesty International said in a report Wednesday.

Hellen Mukiri-Smith, an AI researcher and one of the authors of the report, said "mass surveillance has created a social benefits system that risks targeting, rather than supporting the very people it was meant to protect."

Amnesty has examined four algorithms -- with redacted data -- from among the 60 used by the Danish agency responsible for paying out social benefits, Udbetalning Danmark, to identify fraud over more than a decade.

The algorithms are applied to personal data from public databases of Danish residents, which is allowed under the legislation in the Scandinavian country of 5.9 million inhabitants.

They are used to track down potential fraud in a wide range of areas, from pension payments to parental and sick leave, and student grants.

The data includes information on place of residence, travel, citizenship, place of birth, family relationships and income, which Amnesty noted are "sensitive data points that can also serve as proxies for a person's race, ethnicity, or sexual orientation."

One of the algorithms, dubbed the "Model Abroad," is based in particular on the nationality of the beneficiary, with the aim determining whether people have moved abroad without saying so while still receiving social benefits.

"We argue this does directly violate their right to non-discrimination because of the use of citizenship," David Nolan, another author of the report told AFP.

According to the report, 90 percent of the cases opened as a result of the use of this algorithm turn out not to be fraudulent.

To remedy these shortcomings, Amnesty is calling on the Danish authorities to be more transparent and allow algorithms to be audited.

Feeding personal data into an algorithmic model, in order to identify the risk of fraud, needs to be done with greater care, the organisation argued.

"We want them to ban the use of all kinds of data regarding citizenship or nationality ... which we know discriminate," Nolan added.

Another pitfall of the Danish system's reliance on digital services it that it could also lead to the exclusion of marginalised groups such as the elderly and certain foreigners, as it creates a barrier to access to benefits they could be entitled to, according to Amnesty.

The use of artificial intelligence and algorithms by social services in Western countries has been the subject of much criticism from rights advocacy groups.

In October, 15 organisations -- including Amnesty -- filed a complaint with French courts over an algorithm used by the French social benefit agency CNAF to detect undue payments.