Content of review 1, reviewed on November 18, 2013
This paper skilfully analyses the reliability of a new rating scale for assessing communication skills. The paper is well written and easy to understand. I have nevertheless a number of concerns.
From a methodological perspective this is a very preliminary report based on a very small sample of raters, GPs and encounters/cases. In terms of Kane’s validity perspective, this report is a very tine piece of evidence in the potentially large chain of validity evidence. With a sample of 21 GP-candidates in two (simulated) encounters reliability estimates are really very difficult to interpret. A larger study is needed in which close attention is paid to the size of rater variance in relation to encounter/case variance. Usually rater variance is a relative small part of the overall variance, and most noise in the measurement stems from variability of performance across encounters. It is important provide clarification on this issue, also for this new instrument, but a larger dataset is needed for that.
From a conceptual standpoint the authors are encouraged to provide more justification for the introduction of this particular new instrument. There are other communication instruments for assessing clinical encounters. Why do we need this one on top of all others? What is the problem with the others? In addition, should we measure communication separate from clinical content? In recent years we have developed a more holistic view on competence and should we continue to measure things separately? There are instruments in the literature judging the whole clinical encounter (with their estimates of reliability) that include communication. So why having another instrument that looks at communication as a separate entity? In more recent literature, evidence is coming forward that communication at senior stages of learning (like with GP trainees of GPs themselves) should not be assessed generically, but should take the clinical context into account. Doctors use communication strategies for specific (clinical) purposes, so those purposes should be taken into account when rating communication.
For example, the scoring example on page 7 of the paper scores “not done” as zero. In some clinical situations “not doing” certain communication strategies may be quite appropriate. So from a conceptual framework: why a new instrument and why assess communication separately and generically? I encourage the authors to provide more justification for this.
Some recent studies that may be informative:
Veldhuijzen W, Ram PM, van der Weijden T, van der Vleuten CPM. (2013) Communication guidelines as a learning tool: an exploration of user preferences in general practice. Patient Educ. Couns., 90(2):213–9.
Essers, G., Kramer, A., Andriesse, B., van Weel, C., van der Vleuten, C., & van Dulmen, S. (2013). Context factors in general practitioner-patient encounters and their impact on assessing communication skills-an exploratory study. BMC family practice, 14(1), 65.
Essers, G., van Dulmen, S., van Es, J., van Weel, C., van der Vleuten, C., & Kramer, A. (2013). Context factors in consultations of general practitioner trainees and their impact on communication assessment in the authentic setting. Patient education and counseling.
Salmon, P., & Young, B. (2011). Creativity in clinical communication: from communication skills to skilled communication. Medical education, 45(3), 217-226.
© 2013 the Reviewer (source).