Claes Wohlin
ABSTRACT
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Objective: This paper presents guidelines for conducting literature reviews using a snowballing approach, and they are illustrated and evaluated by replicating a published systematic literature review.
Method: The guidelines are based on the experience from conducting several systematic literature reviews and experimenting with different approaches.
Results: The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.
Conclusions: It is concluded that using snowballing, as a first search strategy, may very well be a good alternative to the use of database searches.
- Kitchenham, B. A., Dybå, T. and Jørgensen, M. 2004. Evidence-based software engineering. In Proceedings of 27th IEEE International Software Engineering Conference, 273--281, IEEE Computer Society, 2004. Google Scholar
Digital Library
- Webster, J. and Watson, R. T. 2002. Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly 26, 2, xiii--xxiii. Google ScholarDigital Library
- Pickard, L., Kitchenham B. A. and Jones P. 1998. Combining empirical results in software engineering. Information & Software Technology 40, 14, 811--821.Google ScholarDigital Library
- Miller, J. 1999. Can results from software engineering experiments be safely combined?. In Proceedings IEEE 6th International Symposium on Software Metrics, 152--158. Google ScholarDigital Library
- Hayes, W. 1999. Research synthesis in software engineering: A case for meta-analysis. In Proceedings 6th IEEE International Software Metrics Symposium, 143--151. Google ScholarDigital Library
- Basili V. R., Shull F. and Lanubile F. 1999. Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25, 4, 456--473. Google ScholarDigital Library
- Kitchenham B. A. and Charters S. 2007. Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, EBSE-2007-01, Keele University.Google Scholar
- Petersen K., Feldt R., Mujtaba S. and Mattsson M. 2008. Systematic mapping studies in software engineering. In Proceedings 12th International Conference on Evaluation and Assessment in Software Engineering. Google ScholarDigital Library
- Kitchenham B. A., Budgen, D. and Brereton, O. P. 2011. Using mapping studies as the basis for further research -- a participant-observer case study. Information and Software Technology 53, 6, 638--651. Google ScholarDigital Library
- Wohlin C. and Prikladnicki R. 2013. Systematic literature reviews in software engineering. Information and Software Technology 55, 6, 919--920. Google ScholarDigital Library
- MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676--687. Google ScholarDigital Library
- Wohlin, C., Runeson, P., da Mota Silveira Neto, P. A., Engström, E., do Carmo Machado, I. and de Almeida, E. S. 2013. On the reliability of mapping studies in software engineering. Journal of Systems and Software 86, 10, 2594--2610.Google ScholarCross Ref
- Kitchenham, B. A., Brereton, O. P., Budgen, D., Turner, M., Bailey, J. and Linkman, S. 2009. Systematic literature reviews in software engineering: A systematic literature review. Information and Software Technology 51, 1, 7--15. Google ScholarDigital Library
- Kitchenham, B. A., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M. and Linkman, S. 2010. Systematic literature reviews in software engineering: A tertiary study. Information and Software Technology 52, 8, 792--805. Google ScholarDigital Library
- Greenhalgh T. and Peacock, R. 2005. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ 331, 7524, 1064--1065.Google Scholar
- Skoglund, M. and Runeson, P. 2009. Reference-based search strategies in systematic reviews. In Proceedings 13th Evaluation and Assessment in Software Engineering, 31-40. Google ScholarDigital Library
- Kitchenham, B. A., Brereton, O. P., Li, Z., Budgen, D. and Burn, A. 2011. Repeatability of systematic literature reviews. In Proceedings of the 15th International Conference on Evaluation and Assessment in Software Engineering, 46--55.Google Scholar
- Zhang, H., Babar, M. A. and Tell, P. 2011. Identifying relevant studies in software engineering. Information and Software Technology 53, 6, 625--637. Google ScholarDigital Library
- Jalali, S. and Wohlin, C. 2012. Systematic literature studies: Database searches vs. backward snowballing. In Proceedings 6th International Symposium on Empirical Software Engineering and Measurement, 29--38. Google ScholarDigital Library
- Jalali, S. and Wohlin, C. 2012. Global software engineering and agile practices: A systematic review. Journal of Software: Evolution and Process 24, 6, 643--659.Google ScholarCross Ref
- Smite, D., Wohlin, C., Galvina, Z. and Prikladnicki, R. 2012. An empirically based terminology and taxonomy for global software engineering. Empirical Software Engineering: An International Journal 19, 1, 105--153. Google ScholarDigital Library
Index Terms
- Guidelines for snowballing in systematic literature studies and a replication in software engineering
Recommendations
Experiences from using snowballing and database searches in systematic literature studies
EASE '15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software EngineeringBackground: Systematic literature studies are commonly used in software engineering. There are two main ways of conducting the searches for these type of studies; they are snowballing and database searches. In snowballing, the reference list (backward ...
Second-generation systematic literature studies using snowballing
EASE '16: Proceedings of the 20th International Conference on Evaluation and Assessment in Software EngineeringSystematic literature studies have become standard practice in software engineering to synthesize evidence in different areas of the discipline. As more such studies are published, there is also a need to extend previously published systematic ...
Systematic literature studies: database searches vs. backward snowballing
ESEM '12: Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurementSystematic studies of the literature can be done in different ways. In particular, different guidelines propose different first steps in their recommendations, e.g. start with search strings in different databases or start with the reference lists of a ...
Comments