Post-review of Garrett, Lazzaro, Ariely and Sharot (2016). The brain adapts to dishonesty. Nature Neuroscience. (Published online on 24 October 2016)

Garrett, Lazzaro, Ariely and Sharot (2016) build upon one of the authors–Ariely–’s ingenious work on dishonesty (see his film “(Dis)Honesty: The truth about life") and claim that they provided “empirical evidence for a gradual escalation of self-serving dishonesty and reveal a neural mechanism supporting it”.

As the authors indicate designing a model to measure dishonesty objectively is extremely difficult, let alone for doing so in an fMRI scanner. Therefore, I applaud the ingenuity and effort that the authors put in order to investigate this important psychological phenomenon.

The article presents two experiments. In both experiments dishonesty was measured as the amount of voluntary overestimation of a quantity (i.e., the total value of money (£) of a number of coins in a jar) in conditions where overestimating wold lead to higher rewards than being accurate. Note that in some blocks of experiment 2 voluntary underestimation was used and no differences between overestimation and underestimation were found.

The participants went through four conditions in the first experiment:

1. Overestimating does not improve the rewards oneself or another participant would receive after the experiment. This is a control condition.
2. Overestimating improves rewards for oneself and reduces those for another participant.
3. Overestimating improves rewards for oneself and that for another participant.
4. Overestimating reduces rewards for oneself and increases the rewards another participant would receive.

Another group of participants went through three conditions in the second experiment: 1. Same control condition as in experiment 1. 2. Overestimating (and in some blocks, underestimating) improves rewards for oneself and it is neutral for another participant. 3. Overestimating (and in some blocks, underestimating) improves rewards for the other participant and it is neutral for oneself.

The magnitude of dishonesty was operationalised by subtracting the estimation provided by each participant in each trial in the control condition from the estimation provided by each participant in each trial in the other conditions that was based in the same stimulus.

Unfortunately, I think the article has a number of serious problems. Of course, this article was published in a prestigious journal and I assume went through tough reviewers, so my assessment could be wrong. I first my concerns, and then explain them in more detail:

1. I am not convinced that the paradigm actually measured dishonesty. However, I will give the benefit of doubt to the authors and accept that they were successful at measuring dishonesty.
2. The magnitude of dishonesty does not seem to be strong.
3. The escalation of dishonesty in the self-serving others-harming condition is rather small (about £2).
4. Figures show that an important number of participants show dishonesty levels close to zero. Therefore, dishonesty may be present in only a subset of the participants.
5. The results do not seem to be robust.
6. There was selective reporting of conditions in the fMRI study.

1. Was dishonesty actually measured?

The exact wording of the instructions is not provided in the article. But based on the explanation provided by the authors I don’t see any indication that the participants were encouraged to provide accurate estimations despite the incentives they would receive. Thus, when this encouragement is missing it is possible that the participants (or at least some of them) thought that the goal of the experiment was to maximise their rewards. Therefore, their overestimation is not dishonest, rather they were trying to comply with the rules. At least, some of the participants could have been confused about what they were supposed to do in the experiment. But, let’s assume that my criticism is not sound and that dishonesty was actually what it was measured in the study.

2. How dishonest were the participants?

The participants were told that the quantity to estimate was always in the range £15 to £35, and the maximum value they could have given as an estimate was £99. Thus, the maximum dishonesty they could have exhibited was between £64 (£99-£35) and £84 (£99-£15). In fact, given that in the control condition participants, estimated, on average around £4 points higher than the actual the magnitude. So the maximum overestimation in the other conditions ranged from about £60 to £80. Visual inspection of figures in the supplementary material show that the magnitude of dishonesty, on average, was around £6 in condition 2 (experiment 1), around £11 in condition 3 (exp. 1), around £0 (as expected) in condition 4 (exp. 1), around £6 in condition 2 (experiment 2), and around £3 in condition 3 (exp. 2). To me these amounts of dishonesty are not impressive, and in some conditions could be driven by a few number of participants. Note that in condition 3 (exp. 1) being dishonest benefited both the participant and another participant and they showed, on average, only £11 out of a maximum of £60 to £80.

3. How much does dishonesty escalate?

A thesis of the authors is that dishonesty escalates over trials. So, it is important to estimate how much does it escalate. Visual inspection of Figure 1 shows that dishonesty escalated, on average, from about £6 to about £8 in condition 2 (exp. 1), and from £11 to £14 approximately in condition 3 (exp. 1). For experiment 2 there is only information on standardised values, so I cannot estimate the magnitude of the escalation. However, the standardised values are lower than those in experiment 1. The authors report quite high correlations between dishonesty and trial number (1 to 60) in experiment 1, but this is problematic. First, no consideration for autocorrelation is given. Second, the r coefficient does not provide relevant information because there could be an escalation of just £1 over 60 trials with r=1, and an escalation of £10 pounds with a weaker correlation.

4. How many participants are dishonest?

This is an extremely important issue. It seems to me that in real life people differ in how honest they are. Therefore, it is very important to determine how many participants show dishonesty, not just calculate an average dishonesty in a sample. The average dishonesty of each individual is shown in Supplementary Figure 1, as dots. Visual inspection shows that there are an important number of participants with dishonesty levels close to zero in condition 2 (exp. 1), and that dishonesty varies from about -£12 to £48, but it is not possible to see how many of the participants show dishonesty levels close to zero.

Moreover, individual differences in levels of dishonesty should have been considered in the fMRI analysis because if people with zero dishonesty levels show the same pattern of brain activity than those with high levels of dishonesty, then that activation pattern has nothing to do with dishonesty. Unfortunately, the fMRI analysis is reported at the group level.

5. Are the results robust?

The results of experiment 2 are not in line with those in experiment 1. (And I applaud the authors for showing these results. It shows they are honest!!)

It is expected that the participants would be more dishonest in condition 2 (exp. 2) than in condition 2 (exp. 1) because in the former they are not hurting the other participant whereas they do in the latter. However, the average level of dishonesty was very similar in these two conditions. Moreover, there were no statistical differences between the two conditions in experiment 2. This shows that people were equally dishonest to benefit themselves and to benefit the other participant. This either shows that participants have some degree of altruistic dishonesty or that the results are not so robust.

Although I am not typically too concerned with adjustments for multiple comparisons because I am more interested on the magnitude of the effects, it is striking that there were about 60 p values calculated (without counting the fMRI study) and no adjustments for multiple comparisons were used. In Supplementary material 2 the authors indicate that it was not needed. I don’t claim that there is no justification, but there is case for the authors to make it explicit.

6. Why not reporting the results of the four conditions in the fMRI study?

In the fMRI study the authors decided not to show the results of condition 1 and 4 (experiment 1). I think both conditions would have been great control conditions. If they showed the same pattern of brain activity as the self-serving other-harming condition the conclusions would have been invalidated.