Content of review 1, reviewed on April 12, 2024
Dear authors,
With pleasure I have read the paper “Causal inference methods reveal weak reciprocal relationships between productivity and plant biodiversity in managed grasslands”. The paper explores the much debated relationship between biodiversity and productivity and aim to contributes to this in two ways: Firstly, by measuring productivity and biodiversity over time, and hence disentangling this relationship by assuming a time-lagged causal relationship between productivity and biodiversity and biodiversity and productivity. Secondly, they use a not so commonly used technique used in econometric research introduced by Dee et al 2022 (Nature communications) to estimate causal relationships. The paper is well written and introduces the problem clearly.
I have a number of comments to improve the paper:
1) It would benefit the reader if a clear causal graph was shown in which all assumed causal relationships are shown. The authors could get inspiration from Imai et al 2019 American Journal of political science. I am specifically worried that the assumption of strict exogeneity is violated (fig 2c), but this does not become entirely clear to me from the description.
2) The authors use the word causal inference in the title and I would like to warn the authors for using this word. It is only when all assumptions are met and the relationship is assumed to be causal that you can interpret the regression coefficients as causal effects. However, no prove of causality is given, nor is the hypothesized causal graph tested against data (such as is the case in path modelling or structural equation modelling). Thus, it could be that others patterns of association are observed in the data, that are not accounted for in the graph. Hence, it could be that the graph as such would be rejected by the data when testing for it with formal approaches.
3) To interpret the effects as causal in the two-way fixed estimators approach, the important assumption is that there are no time-varying plot level confounders. Please convince me that this is the case. I mean when you have an interaction between temperature (time varying), soil type and soil C/N (plot varying), nitrogen mineralization rates may become plot and year specific very quickly.
4) The comparison that the authors make between the traditional mixed effects model approach and the econometric approach is not fair. The difference between the two methods are twofold: 1) random effects are replaced by fixed effects (please also write the equation of the mixed effect model in full); 2) in the traditional approach all possible observed confounders are included which were not included in the econometric approach. I say the comparison is not fair because only if the variables are truly believed to be a confounding variable, one would include these in the mixed effect model approach (See Arif & MacNeil 2022 Ecology Letters). If the covariate is both caused by the child and the parent then this covariate should not be included because it is not a confounding variable. It is not clear to me whether the authors has made a critical selection of these so-called confounders, and whether these are truly confounders. It is not convincing to use the econometric approach and leave out observed confounders. The strength of the approach is to correct for unmeasured confounders but that is not to say that known observed confounders should be left out. Finally, the set of confounders that is corrected for in the econometric approach includes the observed confounders (if they are truly confounders). Why is it that the sign flips when using only observed confounders? Is the set of observed confounders a biased subset of all possible confounders, or are some variables thought to be confounders but in reality are not? This warrants discussion. I would guess that if the observed confounders are truly confounders then the difference would not be that big. Therefore I do not agree that the econometric approach is a more advanced causal inference approach (line 328). In fact, mixed effect models, or fixed effects models are commonly used in path analysis techniques (see Shipley et al 2009 Ecology), and so the difference in the results that the authors observe probably has more to do with the set of confounders identified than with the method as such.
5) It would be an interesting analysis if the authors would add the covariates to the econometric approach and remove them one by one at the time (or two at the time) and see when the sign of the effect flips. That probably gives more insight in what is going on and why to two methods yield different results. Next, a more in-depth discussion could take place whether that variable is indeed a confounder.
Source
© 2024 the Reviewer.
References
Karl, A., E., D. L., Alexandra, W., Judith, H., Daniel, P., Gaeetane, L. P., Peter, M., Christian, W., Fons, v. D. P. 2024. Weak reciprocal relationships between productivity and plant biodiversity in managed grasslands. Journal of Ecology.