Basic reporting

The scientific part of the paper is well-written, but I found the non-scientific part, namely the description of the the performance of the magic, difficult to comprehend. Maybe I am simply not familiar with stage magic but it took me a long time to get a sense of what the magician actually did. The inclusion of a video clip will greatly improve the readability of the paper. If that is not possible due to copyright or technical issues, I think the introduction needs to be revised to give more detailed and precise description of the performance. Since I was not familiar with Cups and Balls, I turned to youtube, hoping to see how Penn and Teller perform the Cups and Balls magic. This probably was not such a good idea because Cups and Balls, as performed in Penn and Teller’s various stage shows and television programs, appeared to be not exactly the same Cups and Balls magic studied in the paper (in the youtube clips, the magician does not tilt the transparent cup to make the ball fall into his hand). This is a source of confusion that can be avoided if the introduction provides more background information.

The second paragraph of Introduction describes how the magic was invented. According to the description, the magician was surprised by how effective the illusion was. The falling of the ball distracted the attention of the magician such that he himself did not see the loading of the cup. However, Figure 2 clearly shows that the subjects could reliably detect the loading of the cup, thus showing that the magic was not magical, after all! How can a successful magician be so wrong about the effectiveness of his performance? I’m puzzled. The description of the magician’s invention of the magic, rather than helping to motivate the experiment, only confused this reviewer.

Experimental design

The design of the experiment is appropriate for the research question. I have a few suggestions on the presentation:

1. The description of the experiment lacks some information: a) the running length of the video; b) randomization of the conditions, if any; and c) The exact instruction that was given to the subject.

2. In “Data analysis” section, “placed or removed” was written as “placed OUR removed”.

3. The Results section begins with a summary figure (Figure 2). This is inadequate. I’d like to see how the subject actually performed in the detection task. A few examples of “raw” data (time points of button press in relationship to the time points where the magician placed or removed the balls) should be provided.

4. The subjects were asked to detect two events (removal and placing of the ball) but Figure 2A collapsed them into one single measure (probability of correct report). The analysis should separate the two, because the two events have different significance. The stated goal of the experiment was to determine “whether the falling ball in Penn & Teller's Cups and Balls generated stronger misdirection, as hypothesized by Teller, than alternative manipulations.” In this context, the “loading” of the cup (ie. placement of the ball) more directly reflects the effectiveness of the illusion than the removal of the ball. Recall that in the introduction, the magician was said to be surprised that he did not see the load, thus making the load the more important. Either that, or I have misunderstood the research question.

If the separating Figure 2A into two turns out to be less than insightful, the authors should at least briefly report the results or make a comment on it.

Validity of the findings

The research question is: how effective was the illusion? Figure 2 answers the question: the falling of the ball was not as effective as the magician suspected. However, the authors found a learning effect. The more the subjects viewed the magic, the less effective the illusion was. Given that, the interpretation of Figure 2A becomes complicated. It is still possible that the magician's manipulation was effective, but only for the first time. This question is not addressed in the manuscript. The manuscript therefore fails to give a compelling answer to the research question expressed in Introduction.

This unsatisfactory situation is difficult to avoid given the subject matter. I don't think it makes the findings invalid but the authors should either try to see if it can be addressed in some way from the experimental data, express the research question differently, or acknowledge the problem in discussion.

A second issue is with Figure 2C. The effect is acknowledged in the text but its significance is never commented upon. Since it is difficult to detect the load in the opaque cup condition, how can the no load condition affect the results? What does this mean?

Comments for the author

The research question is: how effective was the illusion? Figure 2 answers the question: the falling of the ball was not as effective as the magician suspected. However, the authors found a learning effect. The more the subjects viewed the magic, the less effective the illusion was. Given that, the interpretation of Figure 2A becomes complicated. It is still possible that the magician's manipulation was effective, but only for the first time. This question is not addressed in the manuscript. The manuscript therefore fails to give a compelling answer to the research question expressed in Introduction.

This unsatisfactory situation is difficult to avoid given the subject matter. I don't think it makes the findings invalid but the authors should either try to see if it can be addressed in some way from the experimental data, express the research question differently, or acknowledge the problem in discussion.

A second issue is with Figure 2C. The effect is acknowledged in the text but its significance is never commented upon. Since it is difficult to detect the load in the opaque cup condition, how can the no load condition affect the results? What does this mean?