The importance of fair algorithms for consumer trust

Tim Draws, Benjamin Timmermans & Monique van Maare
Posted on May 15, 2020

Consumers are increasingly interacting with artificial intelligence (AI) systems that aim to help them navigate through life. This is no different in banking, where so-called robo-advisors have begun to aid consumers with personal finance tasks such as investing, saving, and picking the right mortgage. Because they can perform these tasks without human interference, robo-advisors are extremely cost-efficient and accessible as compared to human financial advisors (van Thiel, Goedee, Reijnders, 2008). This makes digital robo-advisors promising tools for assisting consumers with financial decision-making (Malhotra & Malhotra, 2006).

A robo-advisor needs to fulfill two key criteria in order to be effective. Firstly, it has to make accurate predictions about the outcomes of its user’s prospective financial decisions. Inaccurate predictions would result in wrong advice, which consumers are unlikely to benefit from. Secondly, consumers need to trust the advice of the robo-advisor. The digital tool needs to be trustworthy: any advice is useless if the recipient has no trust in it and does not follow through on it for that reason.

Whereas AI research is constantly pushing the limits of machine learning algorithms by improving their accuracy, little attention has been paid so far to the important role of trust in the interaction between AI and humans. Therefore, it is the second characteristic of an effective robo-advisor – trustworthiness – that we focused on in our TFI research project.

To trust or not to trust... robo-advisors

Trust in robo-advisors cannot be taken for granted. People are still reluctant to hand over control to AI systems, and research has demonstrated that even whilst people initially trust AI systems, their trust deteriorates as soon as the AI visibly makes a wrong prediction (Dietvorst, Simmons & Massey, 2015). A recent example of this is Apple’s credit card PR disaster. Consumers quickly lost trust in in the “smart” credit card after its algorithms turned out to discriminate against female customers by giving them significantly smaller loans compared to male customers.

In our research, we aimed to investigate the influence of AI's "algorithmic fairness" on consumer trust in robo-advisors. For our first study, we recruited 489 participants (51% female, 49% male; average age of 41.9) via the crowdsourcing platform Figure Eight. Our study was conducted entirely online in the month of June in 2019. Participants were recruited from the Figure Eight pool of participants (476).

We introduced our study participants to a hypothetical scenario in which they were offered to use a robo-advisor – referred to as "the AI advisor" -- in their banking app. The AI advisor’s function was said to continuously monitor the user’s financial situation in order to recommend steps to improve the user’s financial health. These recommendations could entail saving strategies or investment tips.

In our experiment, we first showed participants a statistic comparing customers who used the AI advisor with those who didn’t (suggesting that users of the AI advisor generally benefited from using it), to measure the participants’ trust in and perceived benefit from the AI advisor.1 In the next step, each participant was assigned to one of four groups. Participants in each group were shown similar statistics of AI benefits as in the step before, but now split by gender. For one group, the statistic given suggested that male and female customers equally benefited from using the AI advisor. For the other three groups, however, the statistic displayed per group differed in the degree to which men and women faced AI unfairness, suggesting that male customers benefited more than female customers. Again, we measured trust and perceived benefit.

Table 1a
Table 1a.
Figure 1a & 1b
Table 1b.

Two tables shown to participants before the first (Table 1a) and second moment of measurement (Table 1b). Whereas Table 1a contained the same numbers for all participants, Table 1b was adapted depending on the group the participant was placed in. Table 1b was shown to the most extreme unfair group.

Fairness is key for consumer trust in robo-advisors

Our results suggest, in line with previous research, that perceived algorithmic unfairness negatively affected consumer trust. In our statistical analysis we looked at how much our participants’ self-reported trust level changed after they saw the second table containing the statistics split by gender. The level of unfairness shown in this second table had a negative effect on trust for both male and female participants. Whereas trust did not change for participants in the fair scenario, participants in the three unfair scenarios reported decreases in trust. Similarly, participants in the unfair scenarios were less likely to say they would use the AI advisor after seeing the second table compared to participants in the fair scenario.

In general, our male participants reported higher trust in the AI advisor than women in all scenarios, which could reflect higher financial literacy or risk tolerance of men. However, men also reported decreases in trust as the conditions became more unfair. Thus, despite apparently benefiting from the gender bias in the AI advisor, both men and women doubted the trustworthiness of the system.

Perhaps the most interesting finding of this study was that although women’s perceived benefit decreased with trust in the unfair scenarios, men’s perceived benefit was the same in all groups. This means that the decrease in trust that male participants reported was not due to a distrust in the statistics that were shown. Men in all groups believed that they would personally benefit from the AI advisor, but distrusted the system nevertheless when they realized that the statistics exhibited a gender divide.

Future implications

What do these results tell us? Most importantly, they confirm that algorithmic unfairness affects consumer trust negatively and suggest that this holds for both advantaged and disadvantaged groups. Furthermore, it does not seem to be enough to make one subgroup believe that they will benefit from using an AI system despite an apparent unfairness. This study provides evidence that if algorithmic unfairness is present, people will distrust the service even if they believe that they would personally benefit from using it but others would not.

It should be noted that, in the fictional scenario in our experiment, the bank openly exposed the algorithmic unfairness underlying the AI advisor. Real-world companies are currently unlikely to disclose such statistics if algorithmic unfairness is present. However, companies should not rely on the assumption that 'no one will find out'. As the previously mentioned Apple case shows, there are countless ways in which this might happen. Moreover, it’s not unreasonable to suspect that increased consumer activism or new legislation may require this openness going forward, in the same way that food ingredients, both the healthy ones and the ones less so, must be visible on a product label.

Robo-advisors can only benefit consumers if consumers trust them. Our results show that when robo-advisors act visibly unfairly, consumer trust deteriorates. In order to help consumers to make better financial decisions, it is critical to put algorithmic fairness high on the requirements list when building robo-advisors, also when the target audience could benefit disproportionately from the service. Consumers meanwhile should remain critical when putting their faith in robo-advisors and may expect transparency about the reasoning behind decisions or advice being offered.