Frontiers in Psychology

Published by


Interested in reviewing for this journal?
Editors on Publons
Top handling editors on Publons (Manuscripts handled)
Top reviewers on Publons (Manuscripts reviewed in last 12 months)
Endorsed by


  • Question 6

    Please comment on the method. Key elements to consider:

    • objective errors or fundamental flaws in the methodology
    • purpose of new method or technique
    • appropriateness of context
    • comprehensive description of procedures
    • quality of figures and tables

    Reviewer comment

    I should start off by saying that I liked reading this paper, and I applaud the authors approach of highlighting a much-neglected phenomenon that is usually present in b-CFS data whilst simultaneously presenting a solution to potential problems this effect might pose. Nevertheless, I do have some remarks/suggestions regarding the way in which this phenomenon (i.e., the correlation) is contextualized as well as the solution that is being proposed. I leave it up to the authors to consider whether these suggestions are sufficiently important to (considerably) revise the text or not.

    1. The authors' basic starting observation is the correlation between effect size and overall response times as well as trial-to-trial variability. Given that some researchers talk about effect sizes in the standardized sense (i.e., Cohen's d), is there any particular motivation for the authors to use RT differences as effect size? I can imagine the correlations would disappear for standardized effect sizes. That is, when mean RT differences scale with the variance, this implies standardized effect sizes would remain constant. Basically, I am interested in why mean RT differences are considered the appropriate effect size measure in this context.

    2. Related to my previous comment; In the Results section, the authors introduce the correlations between overall RT and the RT difference as well as the within-condition SD and the RT difference. However, only at the end of the "within-condition variability" section, it is mentioned that it is a well-established fact that mean RTs correlate with the SD of RTs (Wagenmakers & Brown, 2007). Only in the discussion this is touched upon again. Personally, I think what we see here is the result of this "law". That is, under the assumption of a constant standardized difference between conditions (i.e., stable Cohen's d at the participant level), the RT difference needs to increase if overall RT and SD of the RT also increase. As can be read in the introduction of Wagenmakers & Brown (2007), not a lot of research has been performed on this linear relationship, but it has been consistently observed. My point is that this "general law of response times" could be a starting point of the observations, rather than being merely mentioned in passing (i.e., only in the General Discussion it is said that this correlation in probably inherent to response time data). If this aspect of RT data is the starting point, the authors could show the correlations that would be predicted from this, and then start to discuss potential problems associated with this aspect of the data. As I highlighted, this is how I interpret the general pattern of results, and I do not know whether the authors agree with me on this aspect.

    3. In the Results section ("Overall response times"), the authors describe that the observed correlation could impose a potential power problem. However, this argument is only spelled out in words, rather than providing a proof or simulation what the impact could be on Type 1 and/or Type 2 errors. I think it would therefore be helpful for future readers to get an idea of how strongly this between-subject variability impacts the results of statistical analyses. A demonstration could be shown just for the simple paired-sample t-test with different levels of between-subject variability and different effect sizes. Alternatively, the authors could point to some references that highlight this aspect and have worked out the details of how this influences the properties of the statistical procedure that is typically used.

    4. In the Results section ("Overall response times"), the authors propose a normalization method that removes the between-subject variability as well as reduces the skewness of the dependent variable. I agree this method removes between-subject variability, but is there any particular motivation to use such a normalization method over normalization through z-scores, for example? In psycholinguistics, researchers sometimes transform individual response time distributions to the same scale using the z-score transformation (see Faust, Mark E.; Balota, David A.; Spieler, Daniel H.; Ferraro, F. Richard, Psychological Bulletin, Vol 125(6), Nov 1999, 777-799). Would this type of transformation yield better or worse performance compared to the normalization based on proportional differences? Again, in this context I think it could be helpful for the future reader to be able to compare the statistical power of the analysis after normalization and before normalization. It is true that the normality assumption is better met after normalization, but it has been shown that of all assumptions normality is least critical in losing statistical power.

    Question 13

    Is prior work properly and fully cited?

    Reviewer comment

    No I was slightly surprised to see the following statement by the authors: "This view seems at odds with conclusions drawn from recent b-CFS studies showing that semantic or conceptual information can drive the conscious access of initially suppressed visual input".

    As the authors are both authorities in the field, I am sure they are aware of the fact that even in the b-CFS literature these observations have been challenged (i.e., there is not necessarily a discrepancy between binocular rivalry and CFS). I think two of my own studies have shown this for scene integration (Moors et al., 2016, Psych Sci) and semantic processing of words (Heyman and Moors, 2014, PLoS ONE). There are other recent b-CFS studies challenging such a view (e.g., Rabovsky et al., 2016).

    Question 14

    Please add here any further comments on this manuscript.

    Reviewer comment

    1. In their Introduction, the authors discuss that b-CFS might be a relatively sensitive measure to uncover differences in processing strength between stimulus conditions because larger response time differences are observed when baseline response times increase. I think the sensitivity would indeed increase only if the variability of the distributions does not scale with the mean of the distribution. That is, in statistical tests we always calculate some form of signal-to-noise ratio. If the mean difference between distributions increases irrespective of the variability of these distributions, this would greatly increase the signal-to-noise ratio. However, if the variability of the distributions increases along with the mean difference (as the authors observe), I would argue the signal-to-noise ratio remains more or less the same, irrespective of the size of the mean difference between distributions, no? So my question is, do larger raw response time differences always imply that a method is more sensitive? I think this question mostly derives from the fact that I often think about effect sizes in the standardized sense (e.g., Cohen's d) rather than the raw RT differences (that, obviously, have a more straightforward interpretation).

    2. In the Introduction, the authors discuss paradigms consisting of manipulations of stimulus content and stimulus context. They say that some content manipulations like face inversion are interesting because only spatial orientation differs, ruling out many low-level confounds. Isn't it the case that the manipulations of context provide an even better low-level control because the invisible stimulus is not changed at all?

    3. Very minor: small typo in the last paragraph (Kanwisher, 2001) rather than (Kaniwsher, 2001).

    Reviewed by
    Ongoing discussion
  • I believe that this paper has a lot of rich content that can impact the theory and practice of vocational counselling if the authors can communicate clearly to the English speaking counselling community the relevance and significance of Confucius philosophy to counselling psychology. I would have no concerns about this manuscript if it were presented as Chinese philosophical counselling for people familiar with Confucius philosophy. However, since the VEC Model was framed within the scientific psychology context and submitted to an English language journal for a wider audience, I do have several concerns.

    Inadequate Empirical Support for VEC Model

    Since the authors claim that the VEC Model incorporates modernism, they should meet the demand for rigorous scientific research, the defining characteristic of modernism. According to Table 1, VEC is supposed to integrate quantitative and qualitative assessment. The authors need to present both types of assessment and indicate how they are integrated in order to support their claims.

    The information regarding methods for this manuscript is far too scanty for me or any other western researcher to ascertain how the qualitative data was collected and how the content analysis was done. The authors adopted Lewin’s three stages and “cyclical-spiral process” of action research. What is this “cyclical-spiral process” and how was it carried out in the present research? Is there any justification why this method was used by the authors rather than the more contemporary participatory action research method?

    I am also not clear what kind of empirical finding supports their claim that the subjects have “learned the VEC concepts from [their] metaphorical stories” and applied these concepts to career choice decisions (pp. 840-845). Is the empirical support based on most of the statements of the participants?

    Inadequate Explanation of Some Chinese Concepts

    The Yin-Yang principle has already been applied to psychology (See Lomas & Ivtzan, 2015; Wong, 2011). Similarly, Wong’s (2012a) dual-system model is also related to Yin-Yang. In psychological terms, Yin refers to more passive, protective, and internalized ways of adaptation, such as avoidance, acceptance, reframing, reappraisal, attunement, harmonizing, and changing oneself in order to accommodate to circumstances. On the other hand, Yang refers to more energetic, proactive, and expansive ways to change and control the environment, achieve a goal, and solve a problem. It would be helpful if the authors can translate the concept of Yin-Yang into such psychological terms so that psychologists in the west can relate to their paper.

    The authors identified six career movements (pp. 340-343). Are these movements related to any existing counselling processes in mainstream psychology? Or are they artificial or contrived ways to fit the scheme of different combinations of Yin-Yang? There needs to be more explanations of why M-realizing means “yang with yin”? Or why M-adapting means “yang into yin”? If the basic psychological functions of these movement are clearly presented, then they will be understood and applied in the English-speaking world.

    Similarly, we also need some explanation of the stories of gian and kum for those who are not familiar with I Chiang. The author also need to make a case for making hexagrams of I Chiang, a divination text, as the foundation for vocation counselling. Would the stories of gian and kum serve as a practical vocational guide, even for people who do not believe the metaphysical assumptions of I Ching? Are these stories universal or meta-narratives? How are they applied to personal narratives? How is the authors’ narrative approach related to Larry Cochran’s (1997) narrative approach and Richard Young and Ladislav Valach’s (2004) constructive approach?

    What I am proposing is that the authors need to connect with the mainstream English literature so that readers can have a better understanding how the Chinese philosophical concepts can be applied to counselling psychology.

    Modernism vs. Postmodernism (PM)

    The authors state that “according to Confucian philosophy, modernism focuses on yang only, post-modernism stresses yin more than yang” (pp. 205-208). There is nothing in Confucian philosophy that speaks directly about modernism and PM; it is more a case of the authors’ interpretation of Confucianism and their characterization of modernism and PM in terms of Yin-Yang, but such a distinction as per Table 1 seems an over-simplification for the two historical movements. As Gergen (2001) wrote in his influential paper on psychological science in a postmodern context, PM allows for multiple voices, including indigenous psychology and traditional psychology. Therefore, strictly speaking, in terms of the history of psychology science, VEC is part of PM.

    For Krumboltz’s HLT eight components (pp. 267-279), half of them can be classified as Yang and another half as Yin. Thus, this is inconsistent with the authors’ characterisation that PM is more about Yin than Yang. Similarly, for Wong’s meaning-therapy, half of the interventions can be considered Yang (striving to achieve a goal, striving towards a certain end value, striving to serve other people, problem-focusing coping, and striving to achieve mastery and excellence) and half Yin (accepting what cannot be changed, living with suffering and transcending suffering, appreciating life, reframing, reappraising). In other words, VEC is not the only model that includes both Yin and Yang.

    It seems contrived to characterize PM as “subjective meaning-making” and non-linear, while VEC is dialectical. As I have indicated earlier, the dialectical principle has also been incorporated in Wong’s meaning therapy, which adopts a holistic and PM perspective (see also Wong 1998, 2012b, 2015a) and clearly integrates quantitative and qualitative research (Wong, 2016a). Meaning therapy recognizes both spirituality and ethical responsibility towards others (Wong, 2016b).

    In view of the above, the authors are left with two options: (a) Making a stronger case why VEC is different from PM or (b) Acknowledging that Wong’s meaning therapy is similar to VEC’s dialectical approach; therefore VEC could also be considered as an indigenous voice of PM. In addition to Wong’s own meaning therapy, existential theory (i.e., Cohen, 2003; Schultze & Miller, 1996) is also an example of PM theory with a strong goal-action component (i.e., modernism). Generally, the authors need to clarify their thinking regarding modernism and PM and their respective impact on vocational counselling. I don’t think Mendes and Moslehuddin (2006) and Samuel and Pryce (2008) really have something to say regarding these issues. I am also wondering why Super’s (1990) highly influential theory of self-concept development and life space is not recognized as a PM theory. In short, I strongly encourage the authors to relate their conceptual analysis and empirical findings to the existing counselling psychology literature in English so that their ideas can be properly understood. I would be happy to answer any questions from the authors in order to facilitate their revision of this manuscript.


    Cochran, L. (1997). Career counselling: A narrative approach. New York, NY: Sage.

    Cohen, B. N. (2003). Applying existential theory and intervention to career decision-making. Journal of Career Development, 29(3), 195-209.

    Gergen, K. J. (2001). Psychological science in a postmodern context. American Psychologist, 56(10), 803-813.

    Lomas, T., & Ivtzan, I. (2016). Second wave positive psychology: Exploring the positive–negative dialectics of wellbeing. Journal of Happiness Studies, 17(4), 1753-1768.

    Schultze, G., & Miller, C. (2004). The search for meaning and career development. Career Development International, 9(2), 142-152.

    Super, D. E. (1990). A life-span, life-space approach to career development. In D. Brown & L. Brooks (Eds.), Career choice and development: Applying contemporary theories to practice (2nd ed.; pp. 197-261). San Francisco, CA: Jossey-Bass.

    Wong, P. T. P. (1998). Meaning-centered counselling. In P. T. P. Wong, & P. Fry (Eds.), The human quest for meaning: A handbook of psychological research and clinical applications (pp. 395-435). Mahwah, NJ: Erlbaum.

    Wong, P. T. P. (2011). Positive psychology 2.0: Towards a balanced interactive model of the good life. Canadian Psychology, 52(2), 69-81.

    Wong, P. T. P. (2012a). Toward a dual-systems model of what makes life worth living. In P. T. P. Wong (Ed.), The human quest for meaning: Theories, research, and applications (2nd ed., pp. 3-22). New York, NY: Routledge.

    Wong, P. T. P. (2012b). From logotherapy to meaning-centered counseling and therapy. In P. T. P. Wong (Ed.), The human quest for meaning: Theories, research, and applications (2nd ed., pp. 619-647). New York, NY: Routledge.

    Wong, P. T. P. (2015a). Meaning therapy: Assessments and interventions. Existential Analysis, 26(1), 154-167.

    Wong, P. T. P. (2016a, August). Meaning-centered approach to research and therapy, second wave positive psychology, and the future of humanistic psychology. The Carl Rogers Award Acceptance Address presented at the Annual Convention of the American Psychological Association, Denver, CO. (Partially supported by the research grant on Virtue, Happiness, and the Meaning of Life from the John Templeton Foundation)

    Wong, P. T. P. (2016b). Self-transcendence: A paradoxical way to become your best. International Journal of Existential Psychology and Psychotherapy, 6(1). Retrieved from

    Young, R. A., & Valach, L. (2004). The construction of career through goal-directed action. Journal of Vocational Behavior, 64(3), 499-514.

    Reviewed by
    Ongoing discussion
  • Reviewed by
    Ongoing discussion
  • Ongoing discussion
  • The authors are using their own translated versions of measures in some populations, without validating the new translations against the original. This has implications for the use of the scale scores in the analysis. If the un-validated translations do not have the same measurement structure, the scale scores will not be representing the same constructs as the original scale scores (or not representing them accurately). This could inflate the risk of Type 1 and 2 errors.

    The authors are using a measure of sleep quality (the PSQI) to assess the convergent validity of the realtime HRQoL measure, as the retrospective HRQoL has previously demonstrated an association. However, the authors are arguing that the retrospective measure of HRQoL was not accurate (due to various biases). Why are the authors attempting to replicate relationships that were seen with a biased measure?

    The authors do not indicate which measures required translation and for which language

    Perceived stress and social class are listed as being measured, but are not mentioned in the analysis, and so it is not clear what purpose they serve in the study.

    The authors state that they will first conduct a PCA on the aggregated data and compare the resulting structure to that of the retrospective measure, but then go on to say that they will also be assessing invariance across assessment methods. Is this not testing the same thing? Whether the structure of the real-time WHOQOL-BREF is the same as that of the retrospective? I'd suggest removing the PCA, as the invariance testing of assessment methods will provide the same information.

    The authors indicate that they aim to assess measurement invariance of the WHOQOL-BREF across countries, but their planned sample size will not be adequate. The process for establishing measurement invariance requires freely estimating all parameters in each group, and then assessing changes in fit when parameters are constrained to be equal between groups. However, 60 individuals per country will not be enough to freely estimate all parameters - the Physical and Mental domains are comprised of 13 items scored on a 5-point likert scale, this would require estimating 13 factor loadings and 13x4=52 thresholds for a total of 65 parameters per group. A group size of 60 is not sufficient to reliably estimate 65 parameters.

    If the authors cannot demonstrate measurement invariance across countries, combining scores from different countries may not be appropriate. However, if previous studies have demonstrated invariance of the WHOQOL-BREF translations, that should be fine.

    Overall, the authors propose an interesting study which will likely provide some very interesting results. There are, however, some methodological issues that I would suggest the authors consider. The use of unvalidated translated measures, and the inability to assess invariance across countries due to sample size could weaken the credibility of any findings. If these main points could be addressed, I feel that the resulting study would be much stronger for it.

    Ongoing discussion