Overall statement or summary of the article and its findings in your own words

The paper and its research questions are mostly well framed. It aims to assess the impact of the Covid-19 pandemic and the subsequent need to work from home for software developers. A few suggestions on what organisations employing these software developers can do to help is also investigated and the findings presented here.

Overall strengths of the article and what impact it might have in your field

AND

Specific comments on weaknesses of the article and what could be done to improve it

The following sections outline the specific strengths and weaknesses as well as suggestions on how to improve them:

Comments on abstract, title, references

Abstract, title and references:

The aim is clear as it is presented as an 'Objective' statement outlining what the researches are seeking to find.

The study's findings are clear as the results indicate the relationship between the COVID-19 pandemic and the developers wellbeing and productivity have had a negative impact. It is also made clear that improving employee wellbeing and home office ergonomics is important for organisations to conduct. The abstract also explains how the researchers performed their study with details given on the methods used and the analysis models employed.

The references are broad in timeline and sources. They are referenced correctly and although some are almost 25 years old, most are relatively recent. One area that is not covered in the reference list is the specific aspects of the emotions women feel who work in the software development space. Most articles referenced about the wellbeing of software developers appear generic in nature and not covering only aspects of wellbeing when look at the female software developer.

The study's title may need some refinement. Part of the title says "Pandemic Programming" however, the study's abstract does not specifically make mention of this aspect of the software developer's job. A software developer has other aspects to their job e.g. attending meetings or writing documentation. If the productivity and wellbeing of a software developer's role as they work from home during the COVID-19 is to be investigated, then it appears incorrect to highlight only one aspect of the software developer's role in the title.

Comments on introduction/background

What the world already knows about this topic is stated and then contrasted with how that knowledge differs from the situation facing the world today - how remote work affects workers when forced upon by an unprecedented crises rather than when organised with prior notice.

The research question is clearly outlined. It does appear that the research question is justified and explanation has also been given as to why this is so - no studies have been done on the effect on wellbeing and productivity when working from home during a pandemic of this magnitude since the World Wide Web was invented.

Comments on methodology

The process of subject selection is clearly articulated. However, it is not clear why the subjects the authors were aiming at were "all sorts of software professionals" when the title and the methodology itself highlight the need to measure the impact of the COVID-19 pandemic on software developers working from home.

The WHO-5 emotional wellbeing index used was applied twice - once four week before the respondents started working from home and another time again when the started working from home. It is not obvious how the WHO-5 index will be considered to be reliable if the respondents are asked to self-assess their wellbeing four weeks prior to working from home owing to the passage of time and natural tendency to forget. The same argument can be made for the Perceived Productivity (HPQ) measure.

The face validity of the HPQ is sound, however construct validity does not appear to be sufficiently met. This is because, the HPQ is " a self-report instrument designed to estimate the workplace costs of health problems in terms of reduced job performance, sickness absence, and work-related accidents-injuries, " (Ref: Kessler, R. C., Barber, C., Beck, A., Berglund, P., Cleary, P. D., McKenas, D., Pronk, N., Simon, G., Stang, P., Ustun, T. B., & Wang, P. (2003). The World Health Organization Health and Work Performance Questionnaire (HPQ). Journal of occupational and environmental medicine, 45(2), 156–174. https://doi.org/10.1097/01.jom.0000052967.43131.51) . The logical relationship between the health problems of employees and the cost to the workplace that this instrument aims to measure doesn't reflect the same relationship between variables the authors are investigating. Therefore, even if the respondents were not aware of the hypotheses, it doesn't seem likely that the HPQ will provide valid insights about measuring perceived productivity about working from home during a pandemic.

Similarly, the content validity of the Disaster Preparedness (DP) measure is to be questioned as this instrument was developed to assess risk perception and disaster preparedness in Immigrants in Canada. It is not clear what the authors mean when they say "We adapted the items to refer specifically to COVID-19. It can be computed by summing the responses or using factor analysis." It is hard to deduce what adaptations were done to cater to the fact that this measure was aimed at immigrants whereas the audience for this study is presumably not just aimed at immigrants but the general population.

In a similar vein, the face validity of the Fear and Resilience (FR) tool makes sense in terms of face validity, but even with the authors remove some "extreme" questions such as asking whether the respondent is experiencing a shortness of breath, the construct validity of the tool doesn't seem strong. This is because the tool seems to be used in hospitals as part of triage where the patient completing the survey is experiencing an acute condition. The fear factors induced by COVID-19 doesn't always present itself as an acute condition and may not be able to capture the level of fear the respondent actually feels (versus when they present in a hospital emergency department).

It is unclear what the face and content validity comments were on the Ergonomics scale the authors developed and sought feedback on these factors on. As the Ergonomics scale was created for this study, the method lacks further detail on the specifics of how these factors were addressed.

Due to the fact that the authors couldn't find any studies that show how organisations support employees during a crisis, they interviewed three developers which they selected (but don't reveal through what method) to brainstorm ideas to then reveal action items. Although the authors mention that Organisation Support is not a construct in this study, it is used to provide action items as suggestions to participants who will then choose whether or not their organisation offers this action item. Since these answers are not used quantitatively, it is unclear what role these responses play in the study.

As mentioned above, the Pilot recruited twelve colleagues, who may not represent the entire world's software developers' mindsets, backgrounds and experiences accurately to perform face and construct validity analysis on to refine the questions posed in the survey. This therefore puts a question mark on the reliability and validity of the development of the adapted questions in the survey.

Finally, with the incentives adopted to conduct sampling, localisation and advertisement, the channels through which they reported the survey as being available seem broad, however it misses some big technical forums - such as Stack Exchange or GitHub community. On a positive note, there does appear to be enough detail in the method description for reproducing the study.

The explanation of the data cleansing exercise was clear and succinct. I am not too familiar with choice of statistical techniques used to conduct the Validity analysis and can't comment on its effectiveness.

The tables and figures in the paper are clearly presented and given a title. However, there are abbreviations used in Figure 2 on the labels in the x-axis. There are a consistent 3 decimal places used in data both in the figures, tables and text.

The text supporting the figures and tables provides a good background context and reasoning why the particular statistical analysis was conducted or variables used. However, as mentioned before, I am not familiar with statistical techniques and their applicability in different situations, so cannot comment on their effectiveness.

Despite my lack of statistical training, I can clearly understand the statistically significant results presented in Figure 5 where the relationships between causal variables, relative strength and direction are illustrated. However, for the purposes of further clarity, it is suggested that in Figure 5, the Home Office Ergonomics bubble be presented to the right of the Change in Perceived Productivity to avoid "crossing arrows".

The Exploratory findings make it clear what the statistically significant results are and it appears that Table 6 provides what the threshold for statistical significance is, however I cannot confirm this due to my inexperience working with statistical analysis of variables.

The Organisational Support section seems to be little less statistically significant as it is based on a single questionnaire. I am not sure the validity of the conclusions drawn in this section could be deemed to be effective. I'm just flagging this here for further peer review by an expert in statistical analysis.

Comments on discussion and conclusions

Section 6.1 - Recommendations seem to jump straight to the answers of the second part of their research question - how can organisations help without first summarising the results of the first question - how COVID-19 affected programmers. They also offer this perspective of how they believe an organisation can help with one specific aspect of helping employees - in terms of ergonomics - "Helping employees improve the ergonomics of their work spaces, in particular, should help. However, micromanaging foot positions, armrests and hip angles is not what we mean by ergonomics" It appears that the authors of the paper did not feel that traditional or existing means of measuring home office ergonomics wouldn't play a significant role in the impact of working from home during a pandemic. They had defined the areas where they would concentrate on when it comes to ergonomics and defined it earlier in the paper, however, it appears that the authors are negating the importance of traditional ergonomics such as foot positions and hip angles. Instead they recommend that organisations may find it beneficial if they ship out noise cancelling headphones or office chairs. It might help if they still clarify that traditional home office ergonomics should not be ignored in preference of purchasing new equipment.

Section 6.2 outlines the limitations of the study and techniques used within the paper clearly. It outlines the models chosen and causality inferred through specific techniques. As I am not an expert in the field of statistical analysis, I don't believe I can offer significant views on the limitations outlined and any impact they have. One small typographical error is present under Internal Validity - the third sentence and after the comma, the capital 'W' needs to be removed.

Section 6.3 describes implications for future research in an articulate manner, however the initial assumption that software engineers only perform "programming" as their main task as defined in the beginning sections plays another role of apparently exaggerating the results of this study of how software engineering practices are affected during the pandemic.

Section 6.4 contains a "Lessons learned" discourse on research methodology which may not be appropriate to include within the conclusions of this paper. For instance, "Second, Google Forms should not be used to conduct scientific questionnaire surveys" was not one of the stated aims of research of this paper. If there is a different section where personal reflections are to be outlined, perhaps this section should be moved there.

Section 7 of the paper outlines the findings and key "take home messages" of the paper well and makes suggestions for future areas of research.