Abstract

In this study, we investigated the validity of a stealth assessment of physics understanding in an educational game, as well as the effectiveness of different game-level delivery methods and various in-game supports on learning. Using a game calledPhysics Playground, we randomly assigned 263 ninth- to eleventh-grade students into four groups: adaptive, linear, free choice and no-treatment control. Each condition had access to the same in-game learning supports during gameplay. Results showed that: (a) the stealth assessment estimates of physics understanding were valid-significantly correlating with the external physics test scores; (b) there was no significant effect of game-level delivery method on students' learning; and (c) physics animations were the most effective (among eight supports tested) in predicting both learning outcome and in-game performance (e.g. number of game levels solved). We included student enjoyment, gender and ethnicity in our analyses as moderators to further investigate the research questions.


Authors

Shute, Valerie;  Rahimi, Seyedahmad;  Smith, Ginny;  Ke, Fengfeng;  Almond, Russell;  Dai, Chih-Pu;  Kuba, Renata;  Liu, Zhichun;  Yang, Xiaotong;  Sun, Chen

Publons users who've claimed - I am an author

No Publons users have claimed this paper.

Contributors on Publons
  • 2 reviewers
  • pre-publication peer review (FINAL ROUND)
    Decision Letter
    2020/06/20

    20-Jun-2020

    Dear Dr. Shute:

    I am pleased to accept your manuscript entitled "Maximizing learning without sacrificing the fun: Stealth assessment, adaptivity, and learning supports in educational games" in its current form for publication in the Journal of Computer Assisted Learning.

    Congratulations!

    My colleague Akshay K Nair (email:JCALedoffice@wiley.com) will contact you again with further details of the procedure from here and, later, a Wiley colleague will deal with production matters. It is hard to predict exactly which issue it will appear in, although our ambition is to keep acceptance-to-printing waits as short as possible.

    In order to make the research we publish in the Journal of Computer Assisted Learning more accessible to the non-expert or lay reader the Editors of the Journal have decided that all manuscripts should also include a “Practitioner Notes” section outlining - in bullet point form - what is currently known about the subject matter, what their paper adds to this, and finally the implications of study findings for practitioners. Please aim to contribute no more than four bullet points per section of approximately 80 characters (i.e., one full sentence) in order to maintain clarity. For more information on this outreach initiative please read Gunter C, Osterrieder A. Genome Biol. 2012 Aug 31;13(8):168. “A modest proposal for an outreach section in scientific publications” (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3491364/).

    However, your article should be available very quickly in the public domain because of our publisher has adopted an " Early View" system - an internet based advanced publication process. This will also allow you
    to make prompt citations of your paper. For further details about the Early View system, go to this link:
    http://jcal.info/online_early/index.htm

    You may be interested to know that you can receive email alerts of JCAL contents pages (6 emails each year) if you are a registered user of Wiley Online Library. If you would like to be sent email alerts please visit https://onlinelibrary.wiley.com/user-registration to login to an existing account or to register as a user.

    It may also interest you that Wiley – the publisher of JCAL - has signed an agreement with Kudos , a web-based service that provides you as author with a free set of tools to explain and share your work for greater usage and impact. If you have registered with Wiley Author Services and opted into the mailing list you will receive the most streamlined experience, such as having your articles linked directly to your Kudos account and receiving personalized email invitations. Otherwise, you can register for a Kudos account at https://www.growkudos.com/sessions/register.

    Thank you again for submitting your work to JCAL. We look forward to seeing it in print shortly.

    Best wishes

    Prof. Paul Kirschner
    Editor-in-Chief, Journal of Computer Assisted Learning

    Reviewer(s)' Comments to Author:

    Journal of Computer Assisted Learning encourages authors to consider uploading their data collection materials to the IRIS database. IRIS is an online repository for data collection materials used for second language research. This includes data elicitation instruments such as interview and observation schedules, language tests and stimuli, pictures, questionnaires, software scripts, url links, word lists, teaching intervention activities, amongst many other types of materials used to elicit data. Please see http://www.iris-database.org for more information and to upload. Any questions, or the materials themselves, may be sent to iris@iris-database.org. Now your article has been formally accepted, your instrument(s) can be uploaded to the IRIS database, with an 'in press' reference. The IRIS team will add page numbers to the reference once they are available.

    Wiley offers authors the option to make their article available to non-subscribers on Wiley Online Library through their OnlineOpen service. This service is also suitable for authors whose funding agency requires grantees to archive the final version of their article. With OnlineOpen, the author, the author's funding agency, or the author's institution pays a fee to ensure that the article is made available to non-subscribers upon publication via Wiley Online Library, as well as deposited in the funding agency's preferred archive. For the full list of terms and conditions, see http://wileyonlinelibrary.com/onlineopen#OnlineOpen_Terms. Any authors wishing to send their paper OnlineOpen will be required to complete the payment form available from our website at: https://authorservices.wiley.com/bauthor/onlineopen_order.asp.

    P.S. – You can help your research get the attention it deserves! Check out Wiley's free Promotion Guide for best-practice recommendations for promoting your work at www.wileyauthors.com/eeo/guide. And learn more about Wiley Editing Services which offers professional video, design, and writing services to create shareable video abstracts, infographics, conference posters, lay summaries, and research news stories for your research at www.wileyauthors.com/eeo/promotion.

    This journal accepts artwork submissions for Cover Images. This is an optional service you can use to help increase article exposure and showcase your research. For more information, including artwork guidelines, pricing, and submission details, please visit the Journal Cover Image page at www.wileyauthors.com/eeo/covers. If you want help creating an image, Wiley Editing Services offers a professional cover image design service that creates eye-catching images, ready to be showcased on the journal cover at www.wileyauthors.com/eeo/design.

    Decision letter by
    Cite this decision letter
    Author Response
    2020/05/25

    REVIEWERS’ COMMENTS AND AUTHORS’ RESPONSES (Revision 2)

    Reviewer: 1

    In the revised version, the authors have taken account of all relevant comments.

    Thanks!
    ~~~~~~~~~~~~~~~~~~~~~~~

    Reviewer: 2

    1. We both agree that flow does not totally equal to enjoyment. However, in the revision, flow is still mentioned in some places, but it is replaced with engagement in other places. This makes the revision not very consistent. In fact, enjoyment rather than flow or engagement is the variable that this study investigated, I would recommend that the authors should just focus on discussing game enjoyment in the manuscript.

    I did a search through the document and changed instances of engagement to enjoyment throughout. I did leave two references to “flow” in the manuscript, but defined it relative to enjoyment, “The flow state happens when one fully engages in a task, loses track of time, and experiences a deep feeling of enjoyment.” I left “flow” in places referencing Csikszentmihalyi’s work.

    1. For my understanding, ANOVA, ANCOVA, and linear regression actually are the same thing. I am not really sure why was a new 3-level variable of physics animations created for data analysis? Especially for the section of effects of physics animations on game enjoyment, I don’t know why ANOVA and regression were both used? Is it only because no difference was found by using ANOVA, but the authors supposed to see differences? I would suggest that there is no need to create a new 3-level variable. Regression would be enough for investigating the effects of physics animations on learning, game performance, and game enjoyment.

    Yes, they are the same. The only reason we included the ANCOVA with the three levels was to explicitly include a group (n = 56) of students who elected to watch NO physics animations. However we agree that this analysis is redundant with the regression results and thus deleted that ANCOVA paragraph (along with the associated “Table 3”). Also note that we additionally removed the ANCOVAs for the performance variables and enjoyment, and used regression analyses for those outcome variables as well.

    1. I would suggest that the subheadings can be consistent. “Gender and learning” can be modified to “learning by gender” and “ethnicity and learning” to “learning by ethnicity”, the same as they are in the section of game enjoyment.

    Fixed. Thanks!



    Cite this author response
  • pre-publication peer review (ROUND 2)
    Decision Letter
    2020/05/25

    25-May-2020

    Dear Dr. Shute

    ‘We recognize that the impact of the COVID-19 pandemic may affect your ability to return your revised manuscript to us within the requested timeframe. If this is the case, please let us know.’

    Thank you for submitting a revised version of the Manuscript ID JCAL-20-098.R1 entitled "Maximizing learning without sacrificing the fun: Stealth assessment, adaptivity, and learning supports in educational games" to JCAL.

    Both original reviewers have seen the manuscript again and their comments are appended to this email.

    You will find that one is content with the revisions and recommends publication. The second reviewer continues to express reservations. I would like you to read the comments of the reviewer who remains uneasy and comment further or edit your manuscript as appropriate.

    (I am particularly concerned that you comment on all the points made as some seem still suspended without a reaction from authors. I will also ask the second reviewer to consider the points made in this more critical review.
    As this is not an area I feel confident to judge myself, you will understand my concern to give reservations a full airing before making a decision. I hope you will be able to provide a reaction as soon as possible in order that we can move the process forward.)

    Please let my colleague Ivy Rose Fernandico (email JCALedoffice@wiley.com) know your intentions in relation to this possible course of action.

    You will need to follow the same procedure as before to upload your second revision to our online submission system.

    To submit your revised manuscript, please take the following steps.

    1) Log into this address: https://mc.manuscriptcentral.com/jcal.

    2) Click on Author Centre, where you will find your manuscript title listed under "Manuscripts with Decisions".

    3) Under "Actions," click on "Create a Revision." (Your manuscript number has been appended to denote a revision.)

    The submit and revise sequence of steps now begins.

    Step (1): You will be shown the Editors decision letter with reviews appended. You should type your response to these reviews in the “response to decision letter” box (of course you may prefer to paste text in here from a word processed file prepared offline).

    Step (2) You will be shown your previous title and abstract, which you may change if necessary.

    Step (3) Is for confirming author identities

    Step (4) Respond to author questions

    Step (5) Browse and label your new manuscript files.

    Click “upload files”. These will be added to the end of the list of “my files” in groups of up to three items.

    IMPORTANT: Your original files are available to you when you upload your revised manuscript. Please delete any files that are no longer required before completing the submission.

    An important reminder:

    Please take this opportunity to check that your paper meets the journal’s style requirements and standards, in particular that the references are listed in the text and at the end of the paper in APA style. Failure to do so will result in delays in publication, should your paper be accepted. Please see http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1365-2729/homepage/ForAuthors.html for further details.

    We are trying to facilitate speedy publication of manuscripts submitted to JCAL so we ask you to upload your revised manuscript as soon as possible. If it will not be possible for you to submit your revision within one month please contact Ivy Rose Fernandico. Liesbeth is also happy to help if you have any other queries.

    Once again, thank you for submitting your manuscript to the JCAL and I look forward to receiving your further revision.

    Yours sincerely

    Prof. Paul Kirschner
    Editor-in-Chief, Journal of Computer Assisted Learning

    Reviewer(s)' Comments to Author:
    Reviewer: 1

    Comments to the Author
    In the revised version, the authors have taken account of all relevant comments.

    Reviewer: 2

    Comments to the Author
    Thanks the authors for working hard on revising the manuscript according to the reviewers’ comments. Regarding the revision, I still have some minor concerns that should be addressed before it can be accepted.
    1. We both agree that flow does not totally equal to enjoyment. However, in the revision, flow is still mentioned in some places, but it is replaced with engagement in other places. This makes the revision not very consistent. In fact, enjoyment rather than flow or engagement is the variable that this study investigated, I would recommend that the authors should just focus on discussing game enjoyment in the manuscript.
    2. For my understanding, ANOVA, ANCOVA, and linear regression actually are the same thing. I am not really sure why was a new 3-level variable of physics animations created for data analysis? Especially for the section of effects of physics animations on game enjoyment, I don’t know why ANOVA and regression were both used? Is it only because no difference was found by using ANOVA, but the authors supposed to see differences? I would suggest that there is no need to create a new 3-level variable. Regression would be enough for investigating the effects of physics animations on learning, game performance, and game enjoyment.
    3. I would suggest that the subheadings can be consistent. “Gender and learning” can be modified to “learning by gender” and “ethnicity and learning” to “learning by ethnicity”, the same as they are in the section of game enjoyment.

    Decision letter by
    Cite this decision letter
    Reviewer report
    2020/05/24

    Thanks the authors for working hard on revising the manuscript according to the reviewers’ comments. Regarding the revision, I still have some minor concerns that should be addressed before it can be accepted.
    1. We both agree that flow does not totally equal to enjoyment. However, in the revision, flow is still mentioned in some places, but it is replaced with engagement in other places. This makes the revision not very consistent. In fact, enjoyment rather than flow or engagement is the variable that this study investigated, I would recommend that the authors should just focus on discussing game enjoyment in the manuscript.
    2. For my understanding, ANOVA, ANCOVA, and linear regression actually are the same thing. I am not really sure why was a new 3-level variable of physics animations created for data analysis? Especially for the section of effects of physics animations on game enjoyment, I don’t know why ANOVA and regression were both used? Is it only because no difference was found by using ANOVA, but the authors supposed to see differences? I would suggest that there is no need to create a new 3-level variable. Regression would be enough for investigating the effects of physics animations on learning, game performance, and game enjoyment.
    3. I would suggest that the subheadings can be consistent. “Gender and learning” can be modified to “learning by gender” and “ethnicity and learning” to “learning by ethnicity”, the same as they are in the section of game enjoyment.

    Reviewed by
    Cite this review
    Reviewer report
    2020/05/03

    In the revised version, the authors have taken account of all relevant comments.

    Reviewed by
    Cite this review
    Author Response
    2020/04/28

    Reviewers’ Comments and Our (Author) Responses

    Reviewer 1
    (1) For me, the game levels and learning supports all belong to game design issue. This study treated them as two independent factors to run the analyses. Would it be possible that there have interactions between the design of game levels and learning supports? I mean maybe for different game level design, learning supports will have different impact on learning and enjoyment?

    Great point. Towards that end, we included the following new analysis/section to address this issue. “Interaction of Learning Supports and Game Level Difficulty. For our final analysis, we wanted to explore any interactions between viewing learning supports and game levels—specifically game level difficulty. We expected that students would tend to access learning supports (particularly physics animations, hints, and worked examples) when playing the more difficult game levels. To test this hypothesis, we computed the average time spent playing each game level (i.e., average duration) which significantly correlated (r = .56, p < .001) with our composite difficulty index (i.e., the sum of our two game mechanics and physics difficulty indices, ranging from 1 to 10). On average, it took students longer to solve harder compared to easier levels. The average duration of playing game levels significantly correlated with accessing learning supports—i.e., physics animations (r = .29, p = .01), hints (r = .83, p < .001), and worked examples (r = .86, p < .001). These correlations suggest that students were, in fact, accessing more learning supports (both content- and game-related) in more difficult levels.

    (2) I feel a bit confused about in this study did learning supports serve as dependent variables or independent variables? In this study, some of the analyses and results used them as dependent variables, however, some used them as independent ones?

    We deleted the “Physics Animations by Game Condition” paragraph where learning support (i.e., physics animations) existed as the dependent variable, because this analysis was not closely related to our research question pertaining to learning supports (and yes, it was confusing). We also edited throughout the Results section to make the sections link clearly to our research questions.

    (3) To my understanding, enjoyment does not totally equal to flow. I am not really sure if it is suitable to say the in-game learning supports are non-interruptive only according to there was positive correlation between the number of physics animations viewed and the composite measure of game enjoyment. Why wasn’t ANOVA employed to test the difference in game enjoyment?

    We agree that “flow” and “enjoyment” are different. Thus we changed a number of instances of “flow” in the paper to more precisely reflect what we intended. We changed the heading in the discussion from “Effective and Non-Interruptive In-Game Learning Supports” to “Effective In-Game Learning Supports.” We did compute the ANOVA analysis you suggested followed by a regression analysis and reported the results in the Results section with the subheading: “Effects of Learning Supports on Game Enjoyment.” We also summarized the findings in the Discussion section: “Another goal of the current work was to figure out how to maximize learning with a game using various learning supports, without ruining the fun of gameplay. As mentioned earlier, this has been an ongoing challenge in educational games. Our results showed that the best learning support (i.e., physics animations) predicted physics posttest scores, but not at the expense of enjoyment. The number of physics animations viewed also significantly predicted our composite measure of game enjoyment.”

    (4) It is mentioned that the game and learning support satisfaction questionnaire was a 16-item questionnaire. But why were only two items included in the finally analysis? Also, since the second part of the questionnaire was to ask about learning support satisfaction and the example item was “the supports helped me understand the physics.” I think the data analyses of this part could provide evidence showing if the learning supports are only game-level support or support targeting the underlying physics.

    We added an Appendix showing the full list of 16 items comprising the Game and learning support satisfaction questionnaire, and their descriptive statistics. In subsequent sections, we separated the questionnaire into sub-scales to further examine students' specific attitudes such as enjoyment of the game vs. enjoyment of the learning supports.

    (5) On page 13, it would be better to treat pre- and posttest as different variables as well when it comes to the analyses of correlations between the more grained stealth assessment estimates and their associated external measures.

    Good idea! We inserted a new table (Table 1) to report the correlations separately for pretest and posttest as you suggested.

    Reviewer 2
    (1) The authors described some issues of STEM education in the first and second paragraphs of Introduction. However, this paper addresses students’ learning of physics, rather than STEM education. The authors should focus on the topic presented in this paper.

    Thank you for the suggestion. We have made quite a few changes throughout the Introduction to focus more specifically on physics.

    (2) The authors described several reasons why well-designed digital games are suitable vehicles for learning. However, they should also explain why well-designed digital games are suitable vehicles for assessment as assessment is one of the important focuses in this paper.

    We added the following reasons for games-as-assessments: “These games are suitable for assessment because designers can embed continuous measures of learning therein (Authors, 2015b), overcoming problems associated with disruptive and limited assessments evidenced by multiple-choice types of tests (Author, 2013b; Authors, 2019b). For instance, researchers have used games to assess various competencies such as problem solving (e.g., Yang, 2012); computational thinking (Author, 2019d); emotional regulation (Authors, 2019c); visual-spatial abilities and attention (e.g., Green & Bavelier, 2007, 2012; Authors, 2015b); persistence (Author, 2013a); creativity (Jackson et al., 2012); and civic engagement (Ferguson & Garza, 2011). Collectively, the findings show that well-designed digital games can be promising vehicles for assessment and learning (Authors, 2012a; Author, 2016b).”

    (3) The rationale of this study is not clearly stated. There is missing information related to problems/ gaps of existing works and the three research questions examined in this paper.

    We included what “we do not know” after “what we do know” to address the gaps in existing research. Specifically we wrote, “However, we do not currently know the psychometric qualities (i.e., reliability, validity, and fairness) of these in-game assessments, nor do we know the added value of adaptivity in games relative to learning (e.g., Clark, Virk, Barnes, & Adams, 2012; Leemkuil & de Jong, 2012; Sampayo-Vargas, Cope, He, & Byrne, 2013). Finally, incorporating learning supports into a game to maximize learning without losing the fun of gameplay has been an important and unresolved challenge in game-based learning research for over a decade (Author, 2019a; Squire, 2006).”

    (4) The sample included students from different grades (i.e., 9th - 11th grades), who may have different background knowledge on physics understanding. Thus, it is questionable to compare students’ learning outcome with such a sample from different grades. Please clarify this. The authors may consider comparing the learning outcome of students from different grades.

    We agree with you that background knowledge on physics understanding plays an important role in the learning outcome. Consequently, we considered students’ prior knowledge when examining learning outcomes. For example, to compare the learning outcomes by condition, we conducted an ANCOVA with pretest as covariate and posttest as the dependent variable. For most all of the analyses we included pretest as a covariate for that reason (to control for differential incoming knowledge—which showed up for boys vs. girls, as well as ethnicities).

    (5) The authors should present the level’s difficulty for physics understanding in more detail.

    We edited this section as follows (with more information on the physics difficulty index): “Game levels in Physics Playground differ in terms of difficulty. We created two sets of rubrics to determine a level’s difficulty—one concerning its game mechanics, and the other related to the underlying physics. Game mechanics difficulty (which ranged from 1-5 per level) was based on factors such as the relative positions of the ball and the balloon, the number of obstacles present, the novelty of the problem, and the number of objects or parameters required to solve the level. Physics difficulty (which also ranged from 1-5 per level) was based on a rubric created by our physics experts. The rubric considered the primary and secondary physics concepts (see nodes in the middle section of Figure 3) that were associated with each level. Scoring for a level’s primary concept was: force and motion = 0, momentum and energy = 1, torque = 2. Another difficulty point was added if the level: (a) required the balancing of forces (i.e., equilibrium or Newton's third law); (b) involved conservation or the transfer of energy; or (c) consisted of both primary and secondary concepts that came from two different “parents” (e.g., Newton’s first law and energy can transfer). Each level was scored by two raters on both dimensions and any disagreements were resolved in consultation with our physics experts.”

    (6) The validity of the game and learning support satisfaction questionnaire should be reported.

    We added a table in the Appendix containing all of the 16 items and their descriptive statistics. This is intended to give readers a sense about the validity of the questionnaires. We also reported reliabilities for both game satisfaction and learning support satisfaction items: .86 for game satisfaction scale, and .73 for learning supports satisfaction scale.

    (7) The authors should discuss the findings of adaptive sequencing of game levels on learning outcome in more detail. The discussion should also link the findings of the current study with previous relevant studies.

    Thank you for your comment. As suggested, we have elaborated on the discussion governing the findings of adaptive sequencing of game levels in the section of "Adaptivity in level navigation" with specific connections to findings from prior research.



    Cite this author response
  • pre-publication peer review (ROUND 1)
    Decision Letter
    2020/04/20

    20-Apr-2020

    Dear Dr. Shute:

    ‘We recognize that the impact of the COVID-19 pandemic may affect your ability to return your revised manuscript to us within the requested timeframe. If this is the case, please let us know.’

    I am writing in relation to your submission of the Manuscript ID JCAL-20-098 entitled "Maximizing learning without sacrificing the fun: Stealth assessment, adaptivity, and learning supports in educational games" to the Journal of Computer Assisted Learning.

    I have now received feedback from two referees. Their reviews may be found at the bottom of this mailing - although we do sometimes note further comments and advice that might be privately addressed by referees to the Editor. I hope you find the reviews useful. I too have looked at the paper myself and I concur with the remarks that the referees make. You will see that, broadly, they are supportive but they are also critical. I have taken their overall recommendation to be that the paper has potential to be published but it would need revisions and a resubmission before this was appropriate. I am therefore advising that you consider this course of action.

    I hope you will consider revisiting the paper and addressing the comments that are made in these reviews. If you re-submit I would involve our referees again in making a final decision which, of course, at this stage I can not predict. However, we do endeavour to reach these conclusions as speedily as possible. Please let my colleague Ivy Rose Fernandico (email JCALedoffice@wiley.com) know your intentions in relation to this possible course of action.

    You will be unable to make your revisions on the originally submitted version of the manuscript. Instead, revise your manuscript offline using a word processing programme and save it on your computer. Please also highlight the changes to your manuscript within the document by using the track changes mode in MS Word or by using bold or coloured text. Once the revised manuscript is prepared, you can upload it to our online submission site.

    To submit your revised manuscript, please take the following steps.

    1) Log into this address: https://mc.manuscriptcentral.com/jcal.

    2) Click on Author Centre, where you will find your manuscript title listed under "Manuscripts with Decisions".

    3) Under "Actions," click on "Create a Revision." (Your manuscript number has been appended to denote a revision.)

    The submit and revise sequence of steps now begins.

    Step (1): You will be shown the Editors decision letter with reviews appended. You should type your response to these reviews in the “response to decision letter” box (of course you may prefer to paste text in here from a word processed file prepared offline).

    Step (2) You will be shown your previous title and abstract, which you may change if necessary.

    Step (3) Is for confirming author identities

    Step (4) Respond to author questions

    Step (5) Browse and label your new manuscript files.

    Click “upload files”. These will be added to the end of the list of “my files” in groups of up to three items.

    IMPORTANT: Your original files are available to you when you upload your revised manuscript. Please delete any files that are no longer required before completing the submission.

    When submitting your revised manuscript please also upload a "clean" version of your revised paper which does not indicate the changes made to the manuscript.

    Please include with the resubmission a file which contains a full list of the various challenges and suggestions that have been raised by reviewers and/or by myself as Editor. Each item in this list should have attached to it your reaction as implemented in the new manuscript draft. In short, the accompanying file should allow you to declare, locate and explain the edits that you have made in the manuscript. For reviewer comments that have not lead to a change in the manuscript, please include some indication of your reasons for maintaining the status quo. In other words, if you are uneasy about responding to any of the present editorial advice, you should indicate the nature of your reservations.

    Editors are inevitably concerned to protect space in a journal. I should remind you that our recommended length window for single empirical studies is between 4000 and 7000 words. While I do not believe in slavishly following these suggestions, I do feel that economy of writing is usually going to be important in determining whether a paper has impact. I would therefore urge you to carry out any edits with careful attention to the overall balance and length of the text. This may require considering deletions as well as insertions.

    May I also draw your attention to our willingness to publish material that is supplementary to a published report? Such material might include Tables and Figures but also sound and video files. We are able to do this when the material makes a useful elaboration of the published paper although it was not considered necessary to make the main scientific point. Further details are given at this link:
    http://jcal.info/supplementary_material/index.htm. Please contact me if you have any questions about this opportunity.

    An important reminder:

    Please take this opportunity to check that your paper meets the journal’s style requirements and standards, in particular that the references are listed in the text and at the end of the paper in APA style. Failure to do so will result in delays in publication, should your paper be accepted. Please see http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1365-2729/homepage/ForAuthors.html for further details.

    We are trying to facilitate speedy publication of manuscripts submitted to the Journal of Computer Assisted Learning so we ask you to upload your revised manuscript as soon as possible. If it will not be possible for you to submit your revision within two months please contact Ivy Rose Fernandico. Liesbeth is also happy to help if you have any other queries.

    Once again, thank you for submitting your manuscript to the Journal of Computer Assisted Learning and I look forward to receiving your revision.

    Yours sincerely,
    Prof. Paul Kirschner
    Editor-in-Chief, Journal of Computer Assisted Learning

    Reviewer(s)' Comments to Author:
    Reviewer: 1

    Comments to the Author
    This study is well-organized and well-written, I enjoyed reading it a lot. Thanks for giving me the opportunity to review it. Regarding the manuscript, some issues arose from reading it:
    1. For me, the game levels and learning supports all belong to game design issue. This study treated them as two independent factors to run the analyses. Would it be possible that there have interactions between the design of game levels and learning supports? I mean maybe for different game level design, learning supports will have different impact on learning and enjoyment?
    2. I feel a bit confused about in this study did learning supports serve as dependent variables or independent variables? In this study, some of the analyses and results used them as dependent variables, however, some used them as independent ones?
    3. To my understanding, enjoyment does not totally equal to flow. I am not really sure if it is suitable to say the in-game learning supports are non-interruptive only according to there was positive correlation between the number of physics animations viewed and the composite measure of game enjoyment. Why wasn’t ANOVA employed to test the difference in game enjoyment?
    4. It is mentioned that the game and learning support satisfaction questionnaire was a 16-item questionnaire. But why were only two items included in the finally analysis? Also, since the second part of the questionnaire was to ask about learning support satisfaction and the example item was “the supports helped me understand the physics.” I think the data analyses of this part could provide evidence showing if the learning supports are only game-level support or support targeting the underlying physics.
    5. On page 13, it would be better to treat pre- and posttest as different variables as well when it comes to the analyses of correlations between the more grained stealth assessment estimates and their associated external measures.

    Reviewer: 2

    Comments to the Author
    1. The authors described some issues of STEM education in the first and second paragraphs of Introduction. However, this paper addresses students’ learning of physics, rather than STEM education. The authors should focus on the topic presented in this paper.
    2. The authors described several reasons why well-designed digital games are suitable vehicles for learning. However, they should also explain why well-designed digital games are suitable vehicles for assessment as assessment is one of the important focuses in this paper.
    3. The rationale of this study is not clearly stated. There is missing information related to problems/ gaps of existing works and the three research questions examined in this paper.
    4. The sample included students from different grades (i.e., 9th - 11th grades), who may have different background knowledge on physics understanding. Thus, it is questionable to compare students’ learning outcome with such a sample from different grades. Please clarify this. The authors may consider comparing the learning outcome of students from different grades.
    5. The authors should present the level’s difficulty for physics understanding in more detail.
    6. The validity of the game and learning support satisfaction questionnaire should be reported.
    7. The authors should discuss the findings of adaptive sequencing of game levels on learning outcome in more detail. The discussion should also link the findings of the current study with previous relevant studies.

    Decision letter by
    Cite this decision letter
    Reviewer report
    2020/04/20

    1. The authors described some issues of STEM education in the first and second paragraphs of Introduction. However, this paper addresses students’ learning of physics, rather than STEM education. The authors should focus on the topic presented in this paper.
    2. The authors described several reasons why well-designed digital games are suitable vehicles for learning. However, they should also explain why well-designed digital games are suitable vehicles for assessment as assessment is one of the important focuses in this paper.
    3. The rationale of this study is not clearly stated. There is missing information related to problems/ gaps of existing works and the three research questions examined in this paper.
    4. The sample included students from different grades (i.e., 9th - 11th grades), who may have different background knowledge on physics understanding. Thus, it is questionable to compare students’ learning outcome with such a sample from different grades. Please clarify this. The authors may consider comparing the learning outcome of students from different grades.
    5. The authors should present the level’s difficulty for physics understanding in more detail.
    6. The validity of the game and learning support satisfaction questionnaire should be reported.
    7. The authors should discuss the findings of adaptive sequencing of game levels on learning outcome in more detail. The discussion should also link the findings of the current study with previous relevant studies.

    Reviewed by
    Cite this review
    Reviewer report
    2020/04/19

    This study is well-organized and well-written, I enjoyed reading it a lot. Thanks for giving me the opportunity to review it. Regarding the manuscript, some issues arose from reading it:
    1. For me, the game levels and learning supports all belong to game design issue. This study treated them as two independent factors to run the analyses. Would it be possible that there have interactions between the design of game levels and learning supports? I mean maybe for different game level design, learning supports will have different impact on learning and enjoyment?
    2. I feel a bit confused about in this study did learning supports serve as dependent variables or independent variables? In this study, some of the analyses and results used them as dependent variables, however, some used them as independent ones?
    3. To my understanding, enjoyment does not totally equal to flow. I am not really sure if it is suitable to say the in-game learning supports are non-interruptive only according to there was positive correlation between the number of physics animations viewed and the composite measure of game enjoyment. Why wasn’t ANOVA employed to test the difference in game enjoyment?
    4. It is mentioned that the game and learning support satisfaction questionnaire was a 16-item questionnaire. But why were only two items included in the finally analysis? Also, since the second part of the questionnaire was to ask about learning support satisfaction and the example item was “the supports helped me understand the physics.” I think the data analyses of this part could provide evidence showing if the learning supports are only game-level support or support targeting the underlying physics.
    5. On page 13, it would be better to treat pre- and posttest as different variables as well when it comes to the analyses of correlations between the more grained stealth assessment estimates and their associated external measures.

    Reviewed by
    Cite this review
All peer review content displayed here is covered by a Creative Commons CC BY 4.0 license.