You are here: Home / The IPN / News / Recognizing and eliminating stumbling blocks...

Recognizing and eliminating stumbling blocks...

September 16th, 2021

What to consider when conducting a systematic literature review.

Carola Garrecht & Till Bruckermann

In research, a thorough literature review is crucial for the planning of projects as well as for the discussion of results. A systematic literature review, on the other hand, can be used to answer a clearly defined research question based on published literature. For this, one has to collect all available relevant articles. This may sound simple, but it can present some stumbling blocks. Two postdocs working at the IPN report on their experiences.

The procedure of a systematic literature review is quickly explained: Having developed a research question, suitable search terms have to be defined for the search in literature databases. The search should identify all articles relevant to answering the research question in order to bring them together in a literature base, taking into account selection criteria. Therefore, the definition of search terms and the selection of suitable articles are iterated until an exhaustive literature base is reached. Transparent documentation of these steps is the foundation of a solid literature review.

We have compiled some of the stumbling blocks that can occur during this process, based on our experience, and outlined possible solutions.

 

1. Literature search preparation

Work step: Define research question

Challenge: "I want to do a literature review. Can I even answer my research question by doing a literature review?"

Approach: To define a suitable research question for a literature review, one should already have an overview of the current state of research. Literature reviews are suitable if sufficient articles on the research topic have already been published. Experiences from a previous study of the research topic (e.g., in a previous qualification thesis) or a preceding thorough literature review facilitate the clarification of the research question.

 

Work step: Selection of suitable literature databases

Challenge: "There are a variety of literature databases. Which is the right one to answer my research question?"

Approach: To begin with, one focuses on the subject area from which the research question originates. The database ERIC (Educational Resources Information Center), for example, specializes in English-language articles in educational research. FIS Bildung also searches for German-language articles in this field. Subsequently, the search can be extended to non-subject-specific databases such as the Web of Science. In addition, considerations of what type of literature one is looking for help (e.g., journal articles, conference papers, teaching outlines). GoogleScholar, for example, provides a broader search and includes books and conference papers.

 

Work step: Working with search terms

Challenge: "In an initial search, the selection of search terms generates an unbelievably large number or hardly any articles. What can I do?"

Approach: If research questions are not defined precisely enough, search terms may be chosen too vaguely. This can result in an unmanageable or a relatively small number of entries. One possibility would be to open or close the research question thematically by refining or summarizing the search terms. Little tricks such as the use of quotation marks help. This allows to find exact word contexts, for example "climate change" instead of "climate" and "change". In addition, the use of logical operators such as AND and/or OR helps to include or exclude correlations, as does the targeted use of synonyms.

 

Work step: Merging search results

Challenge: "Combining different databases results in complete chaos. How do I resolve this?"

Approach: The use of literature management programs such as Citavi or Endnote is a good structuring aid when creating a literature review. Many literature databases offer an export function of the search results so that they can be imported into the literature programs. In Citavi, for example, duplicates are automatically recognized and sorted out.

 

2. Handling of literature found

Work step: Definition of inclusion and exclusion criteria

Challenge: "Searching the database has brought up a whole bunch of different formats of published literature. How do I decide which articles to include?"

Approach: First and foremost, it depends on the research question. Therefore, at the beginning of the search, criteria are defined which the literature should have in order to be included in the analysis. These inclusion and exclusion criteria can, for example, define the language of the articles to be analyzed or also determine whether only articles that have undergone a peer review process are considered. In addition to formal boundaries, the criteria should also set content boundaries, for example, whether only studies that examined students in formal educational contexts are considered.

 

Work step: Selection of suitable articles

Challenge: "I now know criteria by which I distinguish suitable and unsuitable articles. Do I have to read all the articles to select the appropriate ones?"

Approach: The previously defined inclusion and exclusion criteria help to select suitable articles and should always be used as a basis for the search results. First, all articles are skimmed with regard to their fit to the research question (screening). The title and abstract, but also the journal name, provide clues to the fit. During screening, only those criteria should be applied that can be checked on the basis of the title and abstract. All formally suitable articles are subjected to a content review in a further step. Here, the articles remaining after screening are checked in full text for their fit based on the inclusion and exclusion criteria (eligibility).

Important: Checklists such as the PRISMA flowchart can help to systematically work through and document the previous steps: doi:10.1371/journal.pmed1000097

 

Work step: Evaluation of the found literature

Challenge: "My articles are very heterogeneous and complex. How can I compare them with each other now?"

Approach: The use of an analysis form can be used to structure the data collection (e.g. Critical Review Forms: doi: 10.1371/journal.pone.0193060.s003). This helps to extract relevant information and document it in a way that makes comparison between articles clearer. The articles should also be examined with regard to their methodological quality and significance.

Don't forget: A systematic literature review is neither an overview nor a summary of all articles found, but contributes to the current discourse by addressing a gap in the research field through a clearly defined research question.

 

Dos:

Don’ts:

  • Assess evidence in articles,
  • relate articles to each other,
  • classify contradictory and consistent findings,
  • form and compare groupings, and
  • assess potential bias both at the article level and at the review level.
  • Transcription of all articles found,
  • reproduction of the information.