PA2554 Critical appraisal (1.5 ECTS) Assignment 1

1 Introduction

The main objectives of this assignment are:

• Introduction to guidelines for conducting and reporting experiments and case studies.
• Familiarize with experiment and case study research by reading example studies.
• Learn critical appraisal of empirical studies (by analyzing the example studies) with the help of the guidelines and checklists.
•  Understanding and assessment of conformance and consideration of ethical concerns in empirical research. In Assignment 1, your task (in a group of four students ) is to evaluate two empirical studies for scientific rigor and their conformance to ethical research guidelines.

Please follow the instructions outlined below.

2 Instructions

1. Read the following guidelines for conducting/reporting/reviewing experiments and case study research:

a. Experimentation

i. Chapter 2 and 6 of Wohlin et al. Experimentation in software engineering. Springer Science & Business Media, 2012.
ii. Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. ”Reporting experiments in software engineering.” Guide to advanced empirical software engineering. Springer London, 2008.
201-228.

b. Case study research

i. Per Runeson and Martin H¨ost. ‘Guidelines for conducting and reporting case study research
in software engineering.”Empirical software engineering 14.2 (2009): 131-164.

c. Ethics

i. Chapter 2, Section 2.11 of Wohlin et al. Experimentation in software engineering. Springer Science & Business Media, 2012.
ii. Section 3.3, Runeson, Per, and Martin H¨ost. “Guidelines for conducting and reporting case study research in software engineering.” Empirical software engineering 14.2 (2009): 131-164.
iii. J. Singer and N. G. Vinson. Ethical issues in empirical studies of software engineering, IEEE Transactions on Software Engineering 28(12), 1171–1180, 2002.

2. Using the guidelines listed above to thoroughly review the following studies:

a. Chen, X., Usman, M., and Badampudi, D. (2023, June). Using InnerSource for Improving Internal Reuse: An Industrial Case Study. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering (pp. 348-357). Access here.

b. Farooq, S. U., Quadri, S. M. K., and Ahmad, N. (2013, March). A Controlled Experiment to Evaluate Effectiveness and Efficiency of Three Software Testing Methods. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (pp. 493-494). IEEE. Access here.

3 Deliverables

Individually read and review the two papers listed above. Then discuss the individual evaluation in your group, and submit one report (using the answer sheets provided on Canvas) for each of the two papers, presenting the shared view of the group.

4 Additional reading material

Please see the lectures slides and the course memo (available on the Canvas) for further reading material on
case study research and experimentation.