Journal

Systematic Reviews

Field
Medicine
Reviews

93

Interested in reviewing for this journal?
Editors on Publons
Top handling editors on Publons (Manuscripts handled)
Top reviewers on Publons (Manuscripts reviewed in last 12 months)
Endorsed by

Reviews

  • Published in
    Reviewed by
    Ongoing discussion
  • I would like to congratulate the authors on their thorough revision. The protocol is now much easier to read and more clearly motivated. I have no further objections.

    Published in
    Ongoing discussion
  • The authors present an elaborate protocol for a "realist" systematic review and synthesis to untangle issues in the implementation of public health interventions. I must admit that the subject area and methodology are further removed from my areas of expertise than I originally believed, therefore I may not have been able to critically assess all aspects of the protocol.

    The proposed research appears to be interesting and relevant, and the project includes decision makers from public health agencies in Canada, which is encouraging. The protocol is elaborate and appears to be comprehensive. My main concern is that much of the terminology is not clearly defined. Terms such as "realist synthesis" and "middle range theory" are probably not familiar to readers of Systematic Reviews. These basics should be covered in the introduction.

    Minor comments:

    • Introduction: "Systematic reviews of implementation studies have been conducted in several fields [24, 25, 26, 27] but few in public health." If "few" means you didn't find any, please say so. If "few" means "some", please cite specific examples.

    • Introduction: "Instead, realist review ..." - please provide a clear definition of realist review.

    • Methods / Study Purpose: please define "realist program theory" and "integrated knowledge translation process".

    • Methods / Rationale for Using Realist Synthesis: "medicine-based evidence hierarchies" is awkward and seems inaccurate to me. Perhaps "evidence hierarchies developed for medicine"?

    • Methods / Research Plan: end of first paragraph "as completed" -> "was completed".

    • Methods / Research Plan / Step 1 / working together: "Public Health Ontario - a provincial level organization that links practitioners and researchers to scientific intelligence." What does "scientific intelligence" mean?

    • Methods / Research Plan / Step 2: "realist middle range theory" please define.

    • Methods / Research Plan / Step 3: "from a range of North American and European countries that we believe will be most relevant to a range of public health systems" - please clarify your criteria and be more accurate (e.g. Australia is on the list but not in Europe or North America).

    • Methods / Research Plan / Step 3 / First Level Screening: First you state "Articles identified as 'Maybe' will be moved forward for full text screening." and later "All articles selected in this first level screening will be subject to a full text screening." - although these statements are not contradictory the section would be easier to read if they were closer together.

    • Methods / Research Plan / Step 5 / Endnote: "Endnote will allow us to track the reviews, who has reviewed them, and their categorization into subgroups." is confusing, please rephrase.

    • Supplement 1: item 14 "Not applicable for realist synthesis" - this seems overly optimistic, and I remember that appraisal of the relevance and rigour of individual studies is planned. It seems to me that should be mentioned here. Similarly, items 16 & 17 might apply to some realist syntheses, even if you do not plan to address them in your synthesis. You could simply say that no assessment is planned, and/or argue why it is not relevant in this particular case.

    Published in
    Ongoing discussion
  • The authors present a protocol for a review and network meta-analysis of materials and designs of hip prosthetics.

    While the question appears to be interesting and the approach comprehensive, I should note that I have no medical training and can therefore judge only the proposed methods.

    In my opinion the methods are well thought out and adequately described. The only question I still have is whether the pair-wise meta-analysis will also be Bayesian. If the network meta-analysis will rely entirely on contrast-based data (hazard ratios derived from the primary studies), the difference will likely not be noticeable. However if arm-based data with exact likelihoods are used there might be differences between the frequentist and Bayesian settings, especially if the data are sparse or contain zero cells.

    Published in
    Ongoing discussion
  • The author presents a survey of authors of past systematic reviews that included a network meta-analysis, to assess their opinion of the method as well as supporting tools and guidelines. Although the manuscript is of some interest, I feel it is not suitable for publication in its current form. My main objections are:

    • The manuscript is overly long and the writing style is sloppy. Especially the extensive use of quotations in the results section is unnecessary and leads to much duplication between the main text and the quotations. In my opinion quotations are best left to an appendix, and the results section should be shortened dramatically to give only the main results.

    • The limitations mention the response rate as a potential source of bias. However, it does not mention a source of bias that I find more troubling. Only authors of published reviews were contacted. This means that their project has essentially been successful. I would expect that authors of "failed" reviews (that either tried an NMA but did not include it in their publication, or did not publish at all), might have very different opinions.

    Minor comments:

    • Background, first paragraph: "... combine direct and indirect comparison ..." => "... combine direct and indirect evidence ..." would be more appropriate.

    • I believe it is more common to abbreviate network meta-analysis as "NMA" than as "NWMA" (I certainly prefer the former).

    • Background: the distinction between network meta-analysis and mixed treatment comparisons seems unnecessary, especially given that you seem to use network meta-analysis as including pure indirect comparisons as well, in line with the ISPOR guidelines. You might mention the ISPOR definition once early on (NMA = MTC + IC), if at all, but use NMA exclusively from then onwards.

    • It might be useful to report how many respondents had a certain opinion in more cases than is now done.

    • Conclusion: Please rephrase "More widespread use of the methods must be appropriate use."

    Published in
    Ongoing discussion
  • I thank the authors for the thorough revision they have made. I think the reduced word count will help get the points across to more readers, and the quality of the writing has improved as well. I am now quite happy with the manuscript in general, but I do have a few specific comments.

    Major compulsory revisions

    1. Figure 3 needs to be revised. The caption should mention whether the "Combined" effect is a pair-wise combined effect or the estimate from NMA. Based on the "B vs. A" comparison I have assumed that it is a pooled estimate from NMA. In this case, the effects should be transitive, yet this does not appear to be the case. For example A and B are equally effective according to the "B vs A" comparison, yet the effects of "C vs A" and "C vs B" are quite different. In subfigure (B) the pooled effects are the same as in subfigure (A) even though the randomized evidence has drastically shifted - this is not realistic.

    Minor Essential Revisions

    1. The abstract could be made more concise and cohesive. It is now quite verbose and not always easy to follow (see below for an example).

    2. "Network meta-analysis of RCTs and non-randomized comparative cohort studies", paragraph 4: The Bayesian hierarchical model could use some expansion. The current text says some about what it can do, but nothing about what it is.

    Discretionary Revisions

    1. Abstact: "inclusion of non-randomized studies is increasingly common and desirable" - reads as if the inclusion of non-randomized studies is increasingly desirable. It would also be helpful to more closely connect this sentence to "Non-randomized studies can complement RCTs ...", which provides a rationale for why this would be desirable.

    2. "Background", paragraph 2: please break up the final sentence into several.

    3. "Introduction to network meta-analysis", paragraph 1: "because the studies themselves were not randomized" is ambiguous. Although the alternative meaning contradicts the definition of RCT, it could still be confusing.

    4. "Rationale and caveats for including non-randomized comparative cohort studies in NMA", paragraph 3: It would be useful to discuss at least one alternative to the ITT approach and why it may result in a different estimate.

    5. "Network meta-analysis of RCTs and non-randomized comparative cohort studies", paragraph 3: "There is limited research in this area, especially the latter two approaches." Actually there currently seems to be significant interest in this area. It would be fair to say that little has been published so far, though.

    6. "Network meta-analysis of RCTs and non-randomized comparative cohort studies", paragraph 5: "requires considerable time, effort, and costs than including only RCTs." needs to be reformulated.

    7. "Network meta-analysis of non-randomized comparative cohort studies in large distributed data networks", paragraph 3: "more assessable for analysis by others", should that be "more accessible"?

    8. Figure 1/2: The caption could be expanded to explain the content of the subfigures.

    9. Figure 2A, left panel: "True treatment effect small of negligible" - small OR negligible.

    10. Figure 2A, middle/right panel: "less exaggerated" is strange here, because it precedes "exaggerated". Either reverse the order of the panels, or change the wording (e.g. "somewhat exaggerated" vs "greatly exaggerated").

    Level of interest: An article of importance in its field

    Quality of written English: Needs some language corrections before being published

    Declaration of competing interests: I declare that I have no competing interests


    What next?: Unable to decide on acceptance or rejection until the authors have responded to the major compulsory revisions

    Quality of Figures: Original image files are needed in order to check the figures.

    Published in
    Ongoing discussion
  • Reviewer's report: Major compulsory revisions

    1. The authors address an interesting and timely topic, the inclusion of non-randomized studies in network meta-analysis. A few of these studies have already appeared, and a treatment of the assumptions underlying such analyses, as well as the various options available for conducting them, is very welcome. I appreciate the balanced account of assumptions underlying network meta-analysis as well as the impact of confounding. However, as a "Methodology" paper the current account seems rather thin. I would suggest that the authors add much more detail on the methods they describe, as well as worked examples and/or simulation studies to contrast them if they wish to present this as a "Methodology" article. Otherwise, I think the work would be more suited as a "Commentary" paper. I think it could be an excellent commentary on the current state of the art, if my other comments below are addressed.

    2. The manuscript contains many confusing and/or gramatically incorrect sentences. The authors need to carefully proofread their work and correct it to avoid confusion. Examples:

    3. "Conduct of network meta-analysis of non-randomized studies", paragraph 1 - "although the methods have been largely applied to RCTs" does not fit the sentence.

    4. "Conduct of network meta-analysis of non-randomized studies", paragraph 2 - "but with additional risk of bias given these studies have may stil also be limited" does not make sense.
    5. "Conduct of network meta-analysis of non-randomized studies", paragraph 3 - "Some non-randomized study designs and some may be better suited for network meta-analysis than others." does not make sense.
    6. "Comparing and combining findings from non-randomized studies with RCTs", paragraph 1 - "to evaluate the comparative performance nonrandomized studies and" does not make sense.
    7. "Future directions", paragraph 1 - "the likelihood that patients are more similar among compared with inclusion of" does not make sense.

    8. "What is network meta-analysis?", paragraph 1 - "The key assumption underlying any meta-analysis is transitivity of the studies." - it would be more accurate to say the key assumption is "exchangeability of the studies", and the following two sentences do indeed characterize exchangeability, not transitivity. This assumption is also present for pair-wise meta-analysis. The difference is that in pair-wise meta-analysis, the violation of that assumption can only lead to heterogeneity, whereas in network meta-analysis it can lead to non-transitivity of the relative treatment effects (which is loosely the same thing as inconsistency). I think the observations made at the end of the paragraph regarding pair-wise meta-analysis would also be more clear if framed in this way. The work by Jansen & Naci (2013) that the authors cite provides a similar framework (as do several more technical papers by Lu & Ades).

    9. "Conduct of network meta-analysis of non-randomized studies", paragraph 3 - "The network meta-analytic methods chosen will depend on the study designs of non-randomized studies." - This statement is entirely vacuous. The authors do not specify which methods of network meta-analysis are to be considered, nor which characteristics of the design of non-randomized studies should impact the choice, or how. The remainder of the paragraph seems to deal with methods to adjust for confounding in observational studies, which is a different issue.

    10. "Comparing and combining findings from non-randomized studies with RCTs", paragraph 2 - "Alternatively, non-randomized studies could be incorporated as prior information if a Bayesian approach is used." - Please explain how that is different from naive pooling?

    11. "Comparing and combining findings from non-randomized studies with RCTs", paragraph 2 - none of the alternative approaches to incorporating non-randomized evidence are discussed in any detail. This section needs to be expanded greatly to enable the reader to understand the issues at hand. Especially the bias adjustment models need further description.

    12. "Future directions", paragraph 1 - "albeit at a potential cost of reduced precision." - It is unclear to me whether this would result in an undesirable loss of precision, or the (desirable) elimination of double counting.

    13. "Future directions", paragraph 2 - "has used pair-wise meta-analysis to combine data within CNODES" - please explain what this means exactly? Are results from CNODES themselves the result of meta-analysis? Would that not jeopardize the ability to correct for confounding in a consistent manner? What about the possibility of double counting of participants (as mentioned in the previous comment) if pair-wise meta-analysis is used?

    Minor Essential Revisions

    1. "What is network meta-analysis?" paragraph 1 - "all treatments included in the network meta-analysis could have been included in a trial" - would be clearer to say "a single trial" or "the same trial" (especially given that observational studies are also considered). Perhaps it would be useful to explicitly use "jointly randomizable".

    2. "Conduct of network meta-analysis of non-randomized studies", paragraph 1 - "The validity ... comparisons." - this characterization seems better suited to the section introducing network meta-analysis assumptions. I don't have a problem with the reminding the reader of this assumption and that it is especially problematic for non-randomized studies, but this is not the place to introduce the concept.

    3. "Box 1" - "The application of network meta-analysis in non-randomized studies is more complex and less understood than traditional approaches for stakeholders." - the phrase "for stakeholders" doesn't connect well with the remainder of the sentence. Please rephrase. It is also unclear if "tradtional approaches" refers to pair-wise meta-analysis incorporating non-randomized studies, or to network meta-analysis with only randomized studies.

    4. "Box 1" - point 4 under disadvantages seems redundant / overlapping with point 1. Please distinguish them more clearly.

    Discretionary Revisions

    1. "What is network meta-analysis?" - I would prefer a section heading that is not a question.

    Level of interest: An article of importance in its field

    Quality of written English: Not suitable for publication unless extensively edited

    Statistical review: Yes, and I have assessed the statistics in my report.

    Declaration of competing interests: I declare that I have no competing interests


    What next?: Unable to decide on acceptance or rejection until the authors have responded to the major compulsory revisions

    Quality of Figures: Original image files are needed in order to check the figures.

    Published in
    Ongoing discussion
  • Published in
    Reviewed by
    Ongoing discussion
  • Reviewer's report: The authors have thoroughly addressed my comments. I have only one remark.

    Discretionary Revisions

    1. Under "Statistical analysis", the paragraph on the handling of placebo treatments is self-contradictory. Initially, it states that different modalities will be treated as different treatments, and a model combining these will be a sensitivity analysis. However the final sentence indicates the reverse will be true. I think either approach is fine, but this should be clarified.

    Level of interest: An article of outstanding merit and interest in its field

    Quality of written English: Acceptable

    Statistical review: Yes, and I have assessed the statistics in my report.

    Declaration of competing interests: I declare that I have no competing interests.


    What next?: Accept after discretionary revisions

    Quality of Figures: Acceptable.

    Published in
    Ongoing discussion
  • Reviewer's report: Discretionary Revisions

    As I have no medical training, I have judged the protocol mainly on methodological issues. The protocol appears to address a relevant topic, although my ability to judge this is limited. My overall judgment is that the protocol is solid. I do have a number of questions and comments:

    1. The abstract did not mention the trial registries as included in the search, and this seemed a strange omission. However, the search strategy section does say these are included. The abstract should be corrected on this point.

    2. Point 7 of the inclusion criteria should be clarified. Must trials measure all of all-cause, cardiovascular, and cancer mortality, or is one of sufficient?

    3. Point 8 of the inclusion criteria seems overly restrictive. The methods proposed to be used could also make use of other summaries, e.g. hazard ratios.

    4. Point 6 of the exclusion criteria is currently redundant with point 8 of the inclusion criteria, though this may not be the case if my previous comment is addressed.

    5. The test for publication bias needs clarification. Relative risk most often refers to the risk ratio. If risk ratio is indeed intended, please use that term instead. However, The hazard ratio would be more compatible with the overall analysis, whereas the arcsine difference may have nicer properties (http://pubmed.com/17592831). Why not use one of these scales instead?

    6. Statistical analysis / assessment of convergence: convergence and run length are actually somewhat separate issues. The sentence "Posterior summaries will then be obtained from further iterations in each of the three chains, so that the resulting Monte Carlo error is small." seems to conflate the two. It would be better to say "... chains, with a sufficient number of iterations so that the resulting Monte Carlo error is small.".

    7. Statistical analysis / assessment of inconsistency: "... direct and indirect estimates will be compared [32, 33]." The references appear to indicate that the node-splitting method will be used. Please make explicit what method will be used.

    8. Are there any potential differences between placebos matched to different supplements or different modes of administration? Or could there be a difference between placebo and no intervention? Will this possible confounder be checked for in the treatment network?

    9. There aren't any restrictions on the geographical region, genetic background of participants, or language of publication. The authors should comment on whether there is any potential for differences according to region or race, e.g. based on widely differing dietary habits. Will these potential confounders be explored? The introduction and discussion appear to focus on the Western world.

    Level of interest: An article of outstanding merit and interest in its field

    Quality of written English: Acceptable

    Statistical review: Yes, and I have assessed the statistics in my report.

    Declaration of competing interests: I declare that I have no competing interests.


    What next?: Accept after discretionary revisions

    Quality of Figures: Acceptable.

    Published in
    Ongoing discussion