What is a Systematic Review?

‘Systematic’ is, by dictionary definition, ‘a system, plan, or organized method’ that is ‘methodical in procedure or plan’. It is widely assumed that the ‘systematic review’ is the ‘gold standard’ of the Evidence Based Policy Movement.

The assumption then is that if you are going to review the literature on a given topic or subject, that you have come up with a system of doing so, rather than approaching the topic haphazardly.

A systematic review is a paper that summarizes other papers and aims to provide an exhaustive summary of literature relevant to a research question.

The systematic review is an overview of primary studies that used explicit and reproducible methods. 

More and more, however, systematic reviews are investigating what is called ‘grey literature’ or documents on a particular subject that have not been peer reviewed or necessarily published in academic journals.  It is these kinds of documents that can sometimes give you information on the most up to date studies and/or pilot programmes, particularly in disciplines such as healthcare practice, social studies and community programmes.

Advantages of the systematic review include:

·      explicit methods limit bias;

·      conclusions are more reliable;

·      large amounts of information can be assimilated quickly;

·      information delay is reduced;

·      results between studies can be compared;

·      new hypotheses can be generated;

·      and precision is increased.

Typically, the first step of a systematic review is a thorough search of the literature for relevant papers.  Appropriate databases and citation indexes searched, such as Web of Science and PubMed, as well as any individual journals. Next, the titles and the abstracts of the identified articles are checked against pre-determined criteria for eligibility and relevance.

If you plan to tackle a systematic review yourself, your first port of call should be your subject librarian in the University’s library. She or he will be able to guide you, make suggestions about appropriate databases to begin searching, and how to compile a list of search terms that will be not only workable, but produce manageable results.

While systematic reviews are regarded as the strongest form of medical evidence, a review of 300 studies found that not all systematic reviews were equally reliable.

Systematic review is often applied in the biomedical or healthcare context, but it can be applied in any field of research. While many systematic reviews are based on an explicit quantitative meta-analysis or combination of many trial results, there are also qualitative reviews which adhere to the standards for gathering, analyzing and reporting evidence.

Studies suggest that extending searches beyond major databases, perhaps into grey literature, would increase the effectiveness of reviews.

The following are but several examples of recent developments in systematic review that offer promise for more inclusionary and diverse ways of observing, organizing and classifying evidence, particularly qualitative studies, in the social sciences.

1.   The Delphi process is a way of structuring communication among a group of people in order to get their opinions, offer feedback, and offer insights about a course of action.

2.   The ‘Nominal Group’ technique (a physical gathering of participants—unlike the communication process in Delphi technique by questionnaire), on the other hand, works more closely with a brain-storming technique, but also includes private ranking of ideas and tabulation.

3.   Signal and Noise technique is useful for the fact that the process does not eliminate research simply because it is not at a certain level of evidence or if it has certain methodological weaknesses.

4.   Grey literature is comprised of the literature that is not found in peer reviewed journals and is made up of practitioner journal literature, conference papers, books, literature from a range of public, private and voluntary sector bodies, and government publications.

5.   Scoping Studies are devices by which the ‘scope’ and aim of a proposed study are investigated. Traditionally, the scoping study uses a mix of literature review (with a particular purpose of uncovering previous systematic reviews in the field under study) and stakeholder consultation.

6.   Meta-ethnography is a comparative textural analysis of field studies, using three ways to order them: in terms of one another, set against one another or tied to one another. The premise is based in the assumption that there is always a social and theoretical context in which substantive findings emerge; the recovery of this context is the aim of meta-ethnography.

Questions to ask of the systematic review:

1.     How can systematic reviews themselves be evaluated?

2.     Can you find an important question that the review addresses?

3.     Was a thorough search done and were other potentially important sources explored?

4.     Was methodological quality assessed?

5.     Have the results been interpreted with common sense?

6.     In the long run, did finding the systematic reviews on your topic save you time and increase your knowledge?