Systematic Review – Defining the Question

March 18, 2018 Evidence-Based Medicine, Research No comments , , , ,

Eligibility Criteria

The acronym PICO helps to serve as a reminder of the essential components of review question. One of the features that distinguish a systematic review from a narrative review is the pre-specification of criteria for including and excluding studies in the review (eligibility criteria). Eligibility criteria are  a combination of aspects of the clinical question plus specification of the types of studies that have addressed these questions. The participants, interventions and comparisons in the clinical question usually translate directly into eligibility criteria for the review. Outcomes usually are not part of the criteria for including studies: a Cochrane review would typically seek all rigorous studies of a particular comparison of interventions in a particular population of participants, irrespective of the outcomes measured or reported. However, some reviews do legitimately restrict eligibly to specific outcomes.

Screen Shot 2018 03 17 at 7 34 09 PM













The criteria for considering types of people included in studies in a review should be sufficiently broad to encompass the likely diversity of studies, but sufficiently narrow to ensure that a meaningful answer can be obtained when studies are considered in aggregate. It is often helpful to consider the types of people that are of interest in two steps. First, the diseases or conditions of interest should be defined using explicit criteria for establishing their presence or not. Criteria that will force unnecessary exclusion of studies should be avoided. For example, diagnostic criteria that were developed more recently – which may be viewed as the current gold standard for diagnosing the condition of interest – will not have been used in earlier studies. Expensive or recent diagnostic tests may not be available in many countries or settings.

Second, the broad population and setting of interest should be defined. This involves deciding whether a special population group is of interest, determined by factors such as age, sex, race, educational status or the presence of a particular condition such as angina or shortness of breath. Interest may focus on a particular settings such as a community, hospital, nursing home, chronic care institution, or outpatient setting.

The types of participants of interest usually determine directly the participant-related eligibility criteria for including studies. However, pre-specification of rules for dealing with studies that only partially address the population of interest can be challenging.

Any restrictions with respect to specific population characteristics or settings should be based on a sound rationale. Focusing a review on a particular subgroup of people on the basis of their age, sex or ethnicity simply because of personal interests when there is no underlying biologic or sociological justification for doing so should be avoided.


The second key component of a well-formulated question is to specify the interventions of interest and the interventions against which these will be compared (comparisons). In particular, are the interventions to be compared with an inactive control intervention, or with an active control intervention? When specifying drug interventions, factors such as the drug preparation, route of administration, dose, duration, and frequency should be considered. For more complex interventions (such as educational or behavioral interventions), the common or core features of the interventions will need to be defined. In general, it is useful to consider exactly what is delivered, at what intensity, how often it is delivered, who delivers it, and whether people involved in delivery of the intervention need to be trained. Review authors should also consider whether variation in the intervention (i.e., based on dosage/intensity, mode of delivery, frequency, duration etc) is so great that it would have substantially different effects on the participants and outcomes of interest, and hence may be important to restrict.


Screen Shot 2018 03 17 at 9 33 46 PM

Although reporting of outcomes should rarely determine eligibility of studies for a review, the third key component of a well-formulated question is the delineation of particular outcomes that are of interest. In general, Cochrane reviews should include all outcomes that are likely to be meaningful to clinicians, patients, the general public, administrators and policy makers, but should not include outcomes reported in included studies if they are trivial or meaningless to decision makers. Outcomes considered to be meaningful and therefore addressed in a review will not necessarily have been reported in individual studies. For example, quality of life is an important outcome, perhaps the most important outcome, for people considering whether or not to use chemotherapy for advanced cancer, even if the available studies are found to report only survival. Including all important outcomes in a review will highlight gaps in the primary research and encourage researchers to address these gaps in future studies.

Outcomes may include survival (mortality), clinical events (e.g., strokes or myocardial infarction), patient-reported outcomes (e.g., symptoms, quality of life), adverse events, burdens (e.g., demands on caregivers, frequency of tests, restrictions on lifestyle) and economic outcomes (e.g., cost and resource use). It is critical that outcomes used to assess adverse effects as well as outcomes used to assess beneficial effects are among those addressed by a review. If combinations of outcomes will be considered, these need to be specified. For example, if a study fails to make a distinction between non-fatal and fatal strokes, will these data be included in a meta-analysis if the question specifically related to stroke death?

Review authors should consider how outcomes may be measured, both in terms of the type of scale likely to be used and the timing of measurement. Outcomes may be measured objectively (e.g., blood pressure, number of strokes) or subjectively as rated by a clinical, patient, or carer (e.g., disability scales). It may be important to specify whether measurement scales have been published or validated. When defining the timing of outcome measurement, authors may consider whether all time frames or only selected time-points will be included in the review. One strategy is to group time-points into pre-specified intervals to represent “short-term”, “medium-term” and “long-term” outcomes and to take no more than one of each from each study for any particular outcome. It is important to give the timing of outcome measure considerable thought as it can influence the results of the review.

While all important outcomes should be included in Cochrane reviews, trivial outcomes should not be included. Authors need to avoid overwhelming and potentially misleading readers with data that are of little or no importance. In addition, indirect or surrogate outcome measures, such as laboratory results or radiologic results, are potentially misleading and should be avoided or interpreted with caution because they may not predict clinically important outcomes accurately. Surrogate outcomes may provide information on how a treatment might work but not whether it actually does work. Many interventions reduce the risk for a surrogate outcome but have no effect or have harmful effects on clinically relevant outcomes, and some interventions have no effect on surrogate measures but improve clinical outcomes.Screen Shot 2018 03 18 at 6 49 15 PM

Main Outcomes

Once a full list of relevant outcomes has been complied for the review, authors should prioritize the outcomes and select the main outcomes of relevance to the review question. The main outcomes are the essential outcomes for decision-making, and are those that would form the basis of a “Summary of findings” table. “Summary of findings” tables provide key information about the amount of evidence for important comparisons and outcomes, the quality of the evidence and the magnitude of effect. There should be no more than seven main outcomes, which should generally not include surrogate or interim outcomes. They should not be chosen on the basis of any anticipated or observed magnitude of effect, or because they are likely to have been addressed in the studies to be reviewed.

Primary Outcomes

Primary outcomes for the review should be identified from among the main outcomes. Primary outcomes are the outcomes that would be expected to be analyzed should the review identify relevant studies, and conclusions about the effects of the interventions under review will be based largely on these outcomes. There should in general be no more than three primary outcomes and they should include at least one desirable and at least one undesirable outcome (to assess beneficial and adverse effects respectively).

Secondary Outcomes

Main outcomes not selected as primary outcomes would be expected to be listed as secondary outcomes. In addition, secondary outcomes may include a limited number of additional outcomes the review intends to address. These may be specific to only some comparisons in the review. For example, laboratory tests and other surrogate measures may not be considered as main outcomes as they are less important than clinical endpoints in informing decisions, but they may be helpful in explaining effect or determining intervention integrity.

Types of Study

Certain study designs are more appropriate than others for answering particular questions. Authors should consider a priori what study designs are likely to provide reliable data with which to address the objectives of their review.

Because Cochrane reviews address questions about the effects of health care, they focus primarily on randomized trials. Randomization is the only way to prevent systematic differences between baseline characteristics of participants in different intervention groups in terms of both known and unknown (or unmeasured) confounders. For clinical interventions, deciding who receives an intervention and who does not is influenced by many factors, including prognostic factors. Empirical evidence suggests that, on average, non-randomized studies produce effect estimates that indicate more extreme benefits of the effects of health care than randomized trials. However, the extent, and even the direction, of the bias is difficult to predict.

Specific aspects of study design and conduct should also be considered when defining eligibility criteria, even if the review is restricted to randomized trials. For example, decisions over whether cluster-randomized trials and cross-over trials are eligible should be made, as should thresholds for eligibility based on aspects such as use of a placebo comparison group, evaluation of outcomes blinded to allocation, or a minimum period of follow-up. There will always be a trade-off between restrictive study design criteria (which might result in the inclusion of studies with low risk of bias, but which are very small in number) and more liberal design criteria (which might result in the inclusion of more studies, but which are at a higher risk of bias). Furthermore, excessively broad criteria might result in the inclusion of misleading evidence. If, for example, interest focuses on whether a therapy improves survival in patients with a chronic condition, it might be inappropriate to look at studies of very short duration, except to make explicit the point that they cannot address the question of interest.

Scope of Review Question

The questions addressed by a review may be broad or narrow in scope. For example, a review might address a broad question regarding whether anti platelet agents in general are effective in preventing all thrombotic events in humans. Alternatively, a review might address whether a particular anti platelet agent, such as aspirin, is effective in decreasing the risk of a particular thrombotic event, stroke, in elderly persons with a previous history of stroke.

Determining the scope of a review question is a decision dependent upon multiple factors including perspectives regarding a question’s relevance and potential impact; supporting theoretical, biologic and epidemiological information; the potential generalizability and validity of answers to the questions; and available resources.

Evidence-Based Medicine – How to Ask A Question

August 11, 2015 Clinical Trials, Evidence-Based Medicine, Pharmacy Informatics No comments , , , , ,

Foreground questions can be categorized into 5 types, including:

1.Therapy: determining the effect of interventions on patient-important outcomes (symptoms, function, morbidity, mortality, and costs)

2.Harm: ascertaining the effects of potentially harmful agents (including therapies from the first type of question) on patient-important outcomes

3.Differential diagnosis: in patients with a particular clinical presentation, establishing the frequency of the underlying disorders

4.Diagnosis: establishing the power of a test to differentiate between those with and without a target condition or disease

5.Prognosis: estimating a patient’s future course

Clinical questions often spring to mind in a form that makes finding answers in the medical literature a challenge. Dissecting the question into its component parts to facilitate finding the best evidence is a fundamental skill.

One can divide questions of therapy or harm into 4 parts following the PICO framework: patients or population, intervention(s) or exposure(s), comparator, and outcome. For questions of prognosis, you can use 1 of 2 alternative structures. One has only 3 elements: patients, exposure (time), and outcome. An alternative focuses on patient-related factors, such as age and sex, that can modify prognosis: patients, exposure (e.g., older age or male), comparison (e.g., younger age or female), and outcome. For diagnostic tests, the structure we suggest is patients, exposure (test), and outcome (criterion standard).

You need to correctly identify the category of study because, to answer your question, you must find an appropriately designed study. If you look for a randomized trial to inform the properties of a diagnostic test, you will not find the answer you seek.

Different structures or design of studies can investigate different foreground questions. To answer the foreground question that of interest, one should know these different structure or design of clinical studies. Because different study designs can correspond different type of foreground questions, I arrange following discussion due to the type of foreground questions.

Therapy and Harm

To answer questions about a therapeutic issue, we seek studies in which a process analogous to flipping a coin determines participant’s receipt of an experimental treatment or a control or standard treatment: a randomized trial. Once investigator allocate participant to treatment or control groups, they follow them forward in time to determine whether they have, for instance, a stroke or myocardial infarction – what we call the outcome of interest.

When randomized trials are not available, we look to observational studies in which – rather than randomization – clinician or patient preference, or happenstance, determines whether patients receive an intervention or alternative.

Ideally, we would also look to randomized trials to address issues of harm. For most potentially harmful exposures, however, randomly allocating patients is neither practical nor ethical. For instance, one cannot suggest to potential study participants that an investigator will decide by the flip of a coin whether or not they smoke during the next 20 years. For exposures such as smoking, the best one can do is identify observational studies (often sub classified as cohort or case-control studies) that provide less trustworthy evidence than randomized trials.

Differential Diagnosis

For sorting out differential diagnosis, we need a different study design. Here, investigators collect a group of patients with a similar presentation (e.g., painless jaundice, syncope, or headache), conduct an extensive battery of tests, and, if necessary, follow patients forward in time. Ultimately, for each patient the investigators hope to establish the underlying cause of the symptoms and signs with which the patient presented.


Establishing the performance of a diagnostic test (i.e., the test’s properties or operating characteristics) requires a slightly different design. In diagnostic test studies, investigators identify a group of patients among whom they suspect a disease or condition of interest exists (such as tuberculosis, lung cancer, or iron-deficiency anemia), which we call the target condition. These patients undergo the new diagnostic test and a reference standard (also referred to as gold standard or criterion standard). Investigators evaluate the diagnostic test by comparing its classification of patients with that of the reference standard.


A final type of study examines a patient’s prognosis and may identify factors that modify that prognosis. Here, investigators identify patients who belong to a particular group (such as pregnant women, patients undergoing surgery, or patients with cancer) with or without factors that my modify their prognosis (such as age or comorbidity). The exposure here is time, and investigators follow up patients to determine whether they experience the target outcome, such as an adverse obstetric or neonatal event at the end of a pregnancy, a myocardial infarction after surgery, or survival in cancer.