Assessing Evidence and Information

Assessing Medical Information

People have claimed the ability to cure ills since the dawn of time. Some cures are based on science, in that their mode of action and the principles underlying them are known. Some are known empirically to improve health, but we do not fully understand how they work. Other treatments have been shown to have no benefit, while many have never been rigorously tested. Finally, some have been shown to have only a placebo effect, a benefit achieved by suggestion rather than by direct chemical action.

In 2022, MEDLINE indexed almost 1.4 million articles on medical research.¹ To guide medical practice, various agencies now review these publications and propose clinical guidelines based on the assembled evidence. This supports the practice of evidence-based medicine. However, we don’t have guidelines for every condition, so clinicians need to understand the basics of how to review and evaluate medical research articles. This is complicated by the discovery that many of the research results do not agree.² There are various reasons for this, ranging from differences in the study design, to the perspective of the investigator, to unique characteristics of the people in the study, or the way the results were analyzed. No study is perfect, yet the ideal of practising medicine based on evidence demands that clinicians base their decisions on the best available scientific evidence. Because some evidence is flawed, clinicians must be able to judge the validity of published information, and this forms the theme of critical appraisal of the literature. This chapter concerns learning how to think, rather than what to think.

Critical appraisal

Critical appraisal refers to judging the validity of the procedures used in a study to collect data, identifying possible biases that may have arisen, assessing the adequacy of the analysis and completeness of reporting, evaluating the conclusions drawn, and reviewing the study’s compliance with ethical standards of research. Checklists help guide the critical appraisal process, but ultimately, clinicians must use their judgement to assess the quality of a study and its relevance to their particular clinical question.

The first step in critical appraisal is the application of common sense. This was summarised in 1990 as FiLCHeRS, which stands for Falsifiability, Logic, Comprehensiveness, Honesty, Replicability, and Sufficiency.³

Table 5.1: Standards for evaluating information quality under the acronym FiLCHeRS

Evidence-based medicine

Evidence-based medicine (EBM) refers to “the consistent use of current best evidence derived from published clinical and epidemiologic research in management of patients, with attention to the balance of risks and benefits of diagnostic tests and alternative treatment regimens, taking account of each patient’s unique circumstances, including baseline risk, co-morbid conditions and personal preferences”.⁴

In practice, EBM means integrating one’s clinical experience with the best available external clinical evidence from systematic research. The approach was developed in Canada by Dr. David Sackett and colleagues at McMaster University during the 1970s, and is now recognized as a key foundation of medical practice.⁵ Sackett described evidence-based medicine as (1) the process of finding relevant information in medical literature to address a specific clinical problem, (2) the application of simple rules of science and common sense to determine the validity of the information, and (3) the application of the information to answer a clearly formulated clinical question. The aim is to ensure that patient care is based on evidence derived from the best available studies. Sackett argued that the “art of medicine” lies in taking the results of several sources of evidence and interpreting them for the benefit of this particular patient: the opposite of what he called “cookbook medicine” which follows a fixed recipe. The approach has subsequently been applied beyond clinical medicine to propose, for example, evidence-based public health and evidence-based policy making.

Mnemonic: The “5 A”s of evidence-based medicine

Here is a sequence that a clinician may follow in applying evidence-based medicine in deciding how to handle a challenging clinical case:

1. Assess: Recognize and prioritize this patient’s problems.
2. Ask: Construct clinical questions that facilitate efficient searching for evidence on managing the condition in the literature. Questions usually follow the PICO format: describe the Patient; describe the Intervention being considered; with what will it be Compared, and what is the Outcome being sought? For example: “Will a 48 year-old peri-menopausal woman who wears a magnetic wrist bracelet experience fewer night sweats compared to a similar woman who does not wear such a magnet?”
3. Acquire: Gather evidence from quality sources. Librarians are helpful at this stage.
4. Appraise: Evaluate the evidence for its validity, importance, and usefulness (in particular, does it answer your PICO question?)
5. Apply: Apply to the patient, taking account of their preferences and values and of the clinical circumstances.

For more information on the 5 As, please visit: http://jamaevidence.mhmedical.com and http://www.ebm.med.ualberta.ca/

Types of error in Studies

Critical appraisal judges whether shortcomings in a study design or execution could have produced misleading results. Researchers obviously try to eliminate potential errors in their studies, but this can be exceedingly difficult and always increases the study cost. Potential errors are of two broad types: bias (or systematic distortions in the reported results) and random errors. In addition, studies of causation have to address confounding, a challenge in interpreting statistical associations. These types of error are explained in detail later in this chapter, but brief definitions may help the reader at this point: see the Definitions box.

Appraising Scientific Evidence: Qualitative and Quantitative Research

The practitioner of evidence-based medicine must balance scientific evidence derived from studying groups of people against the unique characteristics of this particular patient: blending the science and art of medicine. Similarly, scientific evidence can be drawn from a combination of quantitative and qualitative research. QUALITATIVE RESEARCH (see Glossary) uses non-numerical observations to offer detailed insight into individual cases, and can often address “Why?” questions such as “Why is this patient not adhering to his treatment?”(Optional further detail is in the Here Be Dragons box). QUANTITATIVE RESEARCH methods use data that can be counted or converted into numerical form, and generally address “How?” questions (How effective is this treatment, compared to a placebo?). Table 5.2 summarizes the different purposes of each approach, which most researchers view as complementary, leading to a “mixed methods” approach.

Table 5.2: Comparison of qualitative and quantitative research methods

Qualitative research

Qualitative research “employs non-numeric information to explore individual or group characteristics, producing findings not arrived at by statistical procedures or other quantitative means. Examples of the types of qualitative research include clinical case studies, narrative studies of behaviour, ethnography, and organizational or social studies.”⁶ Applied to public or population health, qualitative methods are valuable in analyzing details of human behaviour. Beyond merely recording facts (did this person obtain an influenza immunization?), qualitative research delves into motivation and personal narratives that offer insights into why.

Qualitative researchers focus on subjective experiences and reject the positivist idea that there exists an objective reality waiting to be discovered. They argue that human experience can be interpreted in many ways, depending on the perspective of the observer, and that that recording experience or motivation can only be partially objective. Qualitative methods are inductive and flexible, allowing interpretations to emerge from the data rather than from a pre-selected theoretical perspective. Just as successive historians may re-interpret historical events, our understanding of diseases and therapies changes with new discoveries.

Like quantitative research, qualitative studies can be pure or applied, but with greater emphasis on the applied – explaining a particular situation. Qualitative data collection methods may be grouped into in-depth interviewing, participant observation, and focus groups (see the Nerd’s Corner box). The data may take the form of words, pictures or sounds – once described as “any data that is not represented by ordinal values.”⁷

Judging the quality of qualitative research

In judging qualitative research you should consider questions such as:

1. Was the design phase of the project rigorous?

Consider the skill and knowledge of the researcher and the completeness of the literature review. Is the research question clear and suited to a qualitative analysis? The researcher should state the perspective from which the data were gathered and analyzed.

2. Was the study rigorous?

The final sampling should represent all relevant groups (all relevant types of patients, both sexes, the full age range, etc.). In qualitative research, sample size is not necessarily fixed, but may continue until no new ideas or concepts emerge, a situation known as saturation. Hence the sampling methods focus on covering the content of responses, rather than numbers of persons. Nor is the interview script completely fixed. Questions need not be uniform, but should respond to the participants’ verbal and non-verbal cues so that the topic can be fully explored. As the project continues, the interview script may evolve in response to the findings of previous interviews.

3. Can I transfer the results of this study to my own setting?

Clinicians reviewing an article must decide if the context and subjects of the study are sufficiently like their own context and patients for the results to be applicable. The results can also be compared to previous literature: How closely does this study corroborate other work? If it corroborates closely, it is likely to be generalizable and, therefore, transferable to a similar context.

Quantitative research

Quantitative studies in medical research are of two broad types: those that count things, such as the numbers of people with different types of cancer, and studies that identify causal influences, such as whether a treatment reliably produces a cure. The following presentation begins with causal studies, and these generally compare different groups, such as people exposed to a risk factor or given a treatment, with others who were not. However, a fundamental challenge is that causality can never be definitively proven; the best a study can do is to show that the results match a series of criteria for inferring a causal relationship.

Criteria for inferring causation

In 1965, Austin Bradford Hill proposed a set of criteria for assessing the causal status of correlations observed in an epidemiological study; he based these in part on Koch’s postulates from the nineteenth century (see Koch’s postulates box, below). The criteria have been revised many times, so you may find a version with different numbers of criteria. Table 5.3 shows a typical example with a commentary on their limits.

Table 5.3: Criteria for inferring a causal relationship in epidemiology

In the end, whether or not a factor is accepted as a cause of a disease always remains open to dispute, especially when it is not possible to obtain experimental proof. There are still defenders of tobacco who can use technical arguments to point out the flaws in the evidence that smoking causes cancer and heart disease. The following sections outline the major types of research design, commenting on their strengths and limitations.

Research designs

Quantitative research uses a variety of study designs, which fall into two main classes: experimental studies (or trials) and observational studies. Figure 5.1 maps the distinctions between these, starting from the exposure or disease being studied, at the top of the diagram.

Experimental (or interventional) studies

As the name implies, these are studies in which the participants undergo some kind of intervention in order to evaluate its impact. The experimental researcher has control over the intervention, its timing and dose or intensity. The study could test a medical or surgical intervention, a new drug, or an intervention to change lifestyle. As the most methodologically rigorous design, experiments are the default choice for providing evidence for best practice in patient management, so our discussion begins with them.

In its simplest form, an experimental study to test the effect of a treatment follows these steps:

1. The researcher formally states the hypothesis to be tested
2. The researcher selects people eligible for the treatment
3. The sample is divided into two groups
4. One group (the experimental, or intervention group) is given the intervention while the other (the control group) is not, and
5. Relevant outcomes are recorded over time, and the results compared between the two groups.

Step 3 leads to a critical distinction, shown at the left of Figure 5.1: the distinction between a randomized controlled trial and non-randomized designs. In the former, people are allocated in step 3 to intervention and control groups by chance alone, while in the latter the choice of who receives the intervention is decided in some other way, such as according to where or in which order they enter the study. There are many types of non-randomized studies and, because the researcher often does not have complete control over the allocation to experimental or control group, they are regarded as inferior to true randomized designs. They are often called quasi-experimental designs (see the Nerd’s corner box).

Randomization removes systematic bias in allocating patients to treatment or control groups, making them comparable; it also allows the valid use of statistical tests, which often assume a random allocation. But the key advantage is that any other factors that could affect the outcome (aka confounding factors) should be equally represented in each study group—including unknown factors such as genetic characteristics that could affect prognosis. An unbiased random allocation should ensure that the only difference between the two study groups is the intervention. The larger the study sample the more confident we can be that other factors really will be equivalent in the two groups, so any difference in outcomes between the groups can be attributed to the intervention. Nonetheless, this remains a matter of probabilities, which is why we need tests of statistical significance. These show the (hopefully very small) likelihood that observed differences between experimental and control groups could have arisen merely by chance.

Randomized controlled trials

The most common experimental design in medical research is the randomized controlled trial (RCT – see Figure 5.2). An RCT is a true experiment in that the investigator controls the exposure and, in its simplest form, assigns subjects randomly to the experimental or control group (which may receive no treatment, or the conventional treatment, or a placebo). Both groups are followed and assessed in a rigorous comparison of their rates of morbidity, mortality, adverse events, etc. RCTs are most commonly used in therapeutic trials but can also be used in trials of prevention. Most commonly, people are randomly allocated to the study groups individually, but groups of people can also be allocated, or even whole communities. RCTs are often conducted across many centres, as illustrated by clinical trials of cancer treatments.

The steps in an RCT are:

1. State the hypothesis in quantitative and operational terms. For example, using the PICO format: “There will be a 10% reduction in self-recorded night sweats among peri-menopausal women who wear a magnetic wrist bracelet, compared to age-matched women who do not wear a wrist magnet.”
2. Select the participants. This step includes calculating the required sample size, setting inclusion and exclusion criteria, and obtaining free and informed consent.
3. Allocate participants randomly to either the treatment or control group; this is normally done using a computer-generated random allocation. Note that there may be more than one intervention group, for example receiving different strengths of magnet in the menopause study. Note also that the control group often receives the standard treatment to which the new one is being compared (here, perhaps, estrogen therapy), or a placebo (such as a wrist band with a fake magnet).
4. Administer the intervention. This is preferably done in a blinded fashion, so that the patient does not know which group she is in. Ideally, the researcher (and certainly the person intervening and monitoring the patient’s response) should also not know which group a given patient is in (this is called a double-blind experiment). This helps to remove the influence of the patient’s and the clinician’s expectations of the treatments, which could bias their assessment of outcomes. Sometimes, a triple-blind approach is used in which neither patient, nor clinician, nor those who analyze and interpret the data (e.g., read x-rays) know which group received the treatment (the groups are merely labeled A or B). This reduces possible bias even further.
5. At a pre-determined time, the outcomes are monitored (e.g., physiological or biochemical parameters, morbidity, mortality, adverse events, functional health status, or quality of life) and compared between the intervention and control groups using statistical analyses. This indicates whether any differences in event rates observed in the two groups are greater than might be expected by chance alone.

While RCTs are regarded as the best research design we have, they do have limitations. They intentionally study the efficacy of a treatment under carefully controlled experimental conditions, which may not show how well the treatment will work in normal clinical practice. “Efficacy” refers to the potential impact of a treatment under the optimal conditions typical of a controlled research setting. “Effectiveness” refers to its impact under the normal conditions of routine practice. For example, in trial conditions the medication may be efficacious because patients are being carefully supervised and know that they are participating in a research project. Unfortunately, in the real world the medication’s effectiveness may be lower because, without supervision, patients may not take all of their medication in the correct dose. A potentially efficacious intervention may also not be efficient enough to put into practice. Breast self-examination has been shown to detect early breast cancer, but only in trial conditions in which women received constant follow-up by trained nurses. This level of intervention was too costly to be put into routine practice.

Furthermore, clinical trials are often conducted on selected populations – for example, men aged 50 to 74 with unstable angina who are current smokers, have no co-morbidity and are willing to participate in a research study. This selectivity limits the extent to which results can be generalized to typical angina patients, i.e., the study’s EXTERNAL VALIDITY. Trial results may also be biased due to attrition if participants in one or other group drop out of the study. Finally, intervention trials, although designed to detect differences in the known and desired outcomes, may not be large enough to reliably detect previously unknown or rare effects.

An adaptation of the RCT is the “N of 1” trial which can have especial value in testing a treatment for a particular patient in a way that avoids most sources of bias.

Ethics of RCTs

Particular ethical issues (see ETHICS in the Glossary) arise in the conduct of medical experiments. A tension may arise between two basic principles: patients have a right to receive an effective treatment (the principle of beneficence), but it is unethical to adopt a new treatment without rigorous testing to prove efficacy and ensure non-maleficence. Therefore, if there is partial evidence that a treatment appears superior, it may be unethical to prove this in a randomized trial as this would entail denying it to patients in the control group. Hence, an RCT can only ethically be applied when there is genuine uncertainty as to whether the experimental treatment is superior; this is termed equipoise. The problem is, genuine equipoise may make it irrelevant to undertake an expensive trial when there is little reason to believe a new medication to be superior. It is also unethical to conduct trials that offer only marginal social value (e.g., studies that benefit the publication record of the researcher more than the health of patients, or studies that double as marketing projects). Naturally, it is also unethical to continue a trial if the treatment is found to be obviously effective or obviously dangerous. Trials are therefore planned with pre-set stopping rules that specify conditions under which they should be prematurely concluded. Ethical considerations mean that many established treatments will probably never be evaluated by a controlled trial:

• Appendectomy for appendicitis
• Insulin for diabetes
• Anaesthesia for surgical operations
• Vaccination for smallpox
• Immobilization for fractured bones, and
• Parachutes for jumping out of airplanes, as the British Medical Journal humorously noted.¹⁰

Phases of intervention studies

Once a new pharmaceutical treatment has been developed, it undergoes testing in a sequence of phases before it can be approved by regulatory agencies for public use. Clinical trials form the third stage within this broader sequence, which begins with laboratory studies using animal models, thence to human testing:

Phase I:     The new drug or treatment is tested in a small group of people for the first time to determine safe dosage and to identify possible side effects.

Phase II:    The drug or treatment is given to a larger group at the recommended dosage to determine its efficacy under controlled circumstances and to evaluate safety. This is generally not a randomized study.

Phase III:  The drug or treatment is tested on large groups to confirm effectiveness, monitor side effects, to compare it to commonly used treatments, and to develop guidelines on the safe use of the drug. Phase III testing normally involves a series of randomized trials. At the end of this phase, the drug may be approved for public use. The “on label” approval may restrict how the drug can be used, for instance for specific diseases or in certain age groups.

Phase IV:  After the treatment enters the marketplace, information continues to be collected to describe its effectiveness on different populations, and especially to detect possible side effects or adverse outcomes. This does not involve a formal RCT and is called post-marketing surveillance; information comes from several sources, such as reports of side effects from physicians and patients, or data on outcomes such as hospital readmissions obtained from computerized information systems. Large numbers may be required to detect rare or slowly developing side effects.

Observational studies

In an observational study, the researcher observes and records what happens to people under exposure conditions either chosen by the person (as with exercise or diet) or that are outside of their control (most social determinants of health). There is often a comparison group of people who were not exposed. The key difference with experimental studies is that the researcher chooses which populations and exposures to study, but does not influence them. As there is no random allocation of exposures, the major problem in inferring causation is that the exposed and unexposed groups may differ on other key factors that may themselves be true causes of the outcome, rather than the characteristics under study (i.e., confounding).

The simplest form of observational study is a descriptive design.

Descriptive studies

Descriptive studies describe how things are: they count the numbers of people who have diabetes, or who smoke, or are satisfied with their hospital care. Such a study uses descriptive statistics to summarize the group results – percentages, mean or median values, and perhaps the variability or standard deviation. The data for a descriptive study may come from a survey questionnaire or from sources such as electronic medical records, or from surveillance systems, describing person, place, and time of disease occurrences (Who? Where? When?) (see Surveillance in Chapter 7). Descriptive studies are commonly used with small, local populations, such as the patients in your practice, and are often used to collect information for planning services. Descriptive studies generally refer to a single point in time – usually the present – and give a cross-sectional, representative picture of the population, although repeated cross-sectional studies can illustrate trends over time, such as the changing number of smokers in your practice. When a study collects information on several variables, it can describe the associations among the variables (for example, is diabetes more common in men or women and does it vary by smoking status?) This can be used to generate hypotheses, which may then be tested in an analytic study.

Analytic studies

The critical distinction between descriptive and analytic studies is that the latter are designed to test a hypothesis, generally concerned with identifying a causal relationship. When an outcome variable, such as heart disease, is studied in relation to an exposure variable such as body weight, the study does more than count: it tests a hypothesis predicting an association between the two. The interest is no longer purely local, as with a descriptive study, but to draw a more general conclusion that will apply to a broader population. Hence, the representativeness of the study sample is crucially important to ensure the external validity or generalizability of the sample results. To describe the level of confidence with which we can draw general conclusions from a sample, we use inferential statistics (see section on Chance errors in sampling, below).

Analytic observational studies vary in terms of the time sequence and sampling procedures used to collect data, and can be of three types: cross-sectional, cohort, or case-control studies, as shown at the bottom of Figure 5.1.

Cross-sectional analytic studies

Cross-sectional studies use a single time-reference for the data collected (e.g., referring to events during the past 2 weeks). A common cross-sectional design is the analytical survey that measures variables to test hypotheses concerning their relationships. For example, in a nationally representative sample a researcher might test the hypothesis that feelings of stress increase the use of medical services. The researcher might ask whether people were under stress in the past two weeks, then whether they had sought care over the same period.

Suppose that this study produced the following result:

Table 5.4: Stress and health care: calculating the association between two variables

Note that the results can be reported in either of two ways:

1. Of those who suffered stress in the last year, 31% (1442/4651) visited their doctor in the last 2 weeks compared with only 19% (2633/13856) of those who did not suffer stress.
2. Of those who visited their doctor, 35% (1442/4075) reported stress in the previous year, compared with 22% (3209/14432) of those who did not visit their doctor.

Either approach is correct. The researcher is free to decide which way to report the results; the study design allows both types of analysis. But all that can be concluded is that there is an association between the two variables. It might be supposed that stress predisposes people to visit their doctor, or could it be that the prospect of a visit to the doctor causes stress? Or maybe something else (a confounding factor such as worrying about an underlying illness?) caused both. This study provides little evidence in support of a causal relationship – merely the apparent association between stress and the doctor visit. The chief weakness of cross-sectional studies is that they cannot show temporal sequence: whether the factor (stress) pre-dated the outcome (doctor visit) (see the causal criteria in Table 5.3).

Descriptive and analytic studies usually sample individual people, but they may alternatively study groups of people, such as city populations. These ecological studies often use aggregate data from government sources, making the study easy to undertake. See the Nerd’s Corner box on Ecological studies.

Cohort studies

A cohort is a group of people who can be sampled and enumerated, who share a defining characteristic and who can be followed over time: members of a birth cohort share the same year of birth, for example (see  the Nerd’s Corner: Roman cohorts). Cohort studies of health commonly study causal factors; the characteristic of interest is usually some sort of exposure hypothesized to increase the likelihood of a disease. A typical cohort study begins with a representative sample of people who do not (yet) have the disease of interest; it collects information on exposure to the factor being studied, then follows exposed and unexposed people over time (Figure 5.3). Because of this, cohort studies are also known as follow-up or longitudinal  studies. The numbers of newly occurring (incident) cases of disease are recorded and compared between the exposure groups. The hypothesis to be tested is generally that more disease will arise in the exposed group (as indicated by the relative sizes of the rectangles on the right of the figure).

In simple cohort studies the results can be fitted into a “2 by 2” table (2 rows by 2 columns – we don’t count the Total column).

Table 5.5: Generic format for a 2 x 2 table linking an exposure to an outcome.

The incidence, or risk, of disease in the exposed group is calculated as a / (a + b). Correspondingly, the risk in the non-exposed people is c / (c + d). These risks can be compared to get a risk ratio (often called a relative risk, or RR): [a/(a + b) divided by c/(c + d)]. This gives an estimate of the strength of the association between the exposure and the outcome: how much more frequent is the disease among those exposed? A relative risk of 1.0 indicates that exposed people are neither more nor less likely to get the disease than unexposed people: there is no association between exposure and disease. A relative risk greater than 1.0 implies that people who have been exposed have a greater chance of becoming diseased, while a relative risk of less than 1.0 implies a protective effect (e.g., a reduced risk of COVID-19 among those who had been immunized).

The main advantage of cohort studies is that exposure was recorded before the outcomes, thus meeting the causal criterion of a temporal sequence between exposure and outcome – as long as study participants truly did not have the disease at the outset. Furthermore, because recording of exposures and outcomes is planned from the beginning of the study period, measurements can be standardized. Note that randomized trials are an experimental version of a cohort study.

Definition of exposure groups

Imagine a cohort study designed to test the hypothesis that exposure to welding fumes causes diseases of the respiratory tract. Whom should you sample? A crude but simple approach would be to base exposure on occupation (assume that metal workers have been exposed; carpenters would be presumed to be unexposed). This approach is frequently used in occupational and military epidemiology. A more precise alternative would be to quantify levels of exposure (e.g., from the person’s work history); this requires considerably more information but would permit dose-response to be estimated—one of the criteria for inferring causation listed in Table 5.3.

In an extension of this quantified approach, a cohort study might not select exposed and unexposed groups to follow, but instead select a representative sample of individuals with sufficient variability in their exposure to permit comparisons across levels of exposure. This allows for mathematical modelling of the effect of exposures. Cohort studies of diet, exercise, or smoking often use this approach, obtaining exposure information from a baseline questionnaire. This approach has been used in community cohort studies such as the Framingham Heart Study (see Illustration: the Framingham Study). Cohort studies offer a powerful way to evaluate causal influences, but they may take a long time to complete and hence be expensive. A cheaper alternative is the case control design.

Case-control studies

Case-control studies compare a group of patients with a particular outcome (e.g., cases of pathologist-confirmed pancreatic cancer) to an otherwise similar group of people without the disease (the controls). As shown in Figure 5.4, reports or records of exposure (e.g., alcohol consumption) before the onset of the disease are then compared between the groups. The name of the design reminds you that groups to be compared are defined in terms of the outcome of interest: outcome present (cases) or absent (controls). The hypothesis to be tested is that exposure will be more common among cases than controls, as suggested by the relative size of the circles on the left of the figure.

Note that in a case-control study you cannot calculate the incidence or risk of the disease, because the study begins with predetermined numbers of people with and without the disease. Therefore, a risk ratio cannot be calculated. But do not despair: you can calculate the odds of a case having been exposed—the ratio of a:c in Table 5.6. This can be compared to the odds that a control was exposed, the ratio of b:d. The result of the case-control study is then expressed as the ratio of these two odds, giving the odds ratio (OR): a/c divided by b/d. To make the calculation easier, this is usually simplified algebraically to ad/bc (i.e., evidence supporting the hypothesis divided by evidence against).

Table 5.6: Generic 2 x 2 table for calculating an odds ratio

The OR calculated from a case-control study can approximate a relative risk, but only when the disease is rare (say, affecting up to around 5% of the population, as in many chronic conditions). The interpretation of the value of an OR is similar to an RR. Like a relative risk, an OR of 1.0 implies no association between exposure and disease. A value over 1.0 implies a greater chance of diseased people having been exposed compared to controls. A value below 1.0 implies that the factor is protective. This might occur, for example, if a case-control study showed that eating a low fat diet protected against heart disease.

Key contrast between cohort and case-control studies

In cohort studies, the participant groups are classified according to their exposure status (whether or not they have the risk factor).

In case-control studies, the different groups are chosen according to their health outcomes (whether or not they have the disease).

Estimating absolute risk: attributable risk and number needed to treat

The RR and OR indicate how much an individual’s risk of disease is increased by having been exposed to a causal factor, in relative terms. Both statistics answer the question a patient who smokes might ask: “Compared to my brother who doesn’t smoke, how much more likely am I to get the disease?” The statistics give the answer as a ratio: “You are twice as likely”, or “10% more likely”. But the patient may equally ask about absolute risk, which relates to disease incidence, as in “What is my chance of getting the disease (in the next year, ten years, over my lifetime)?” The answer is given as an absolute proportion, such as 1 in 10, or 1 in 100. An important point to bear in mind when communicating with a patient is that if the disease is rare, quoting a relative risk of 2 or 3 can appear quite frightening even though the absolute risk is small. A risk factor that doubles an absolute risk of 1 in a million is still only 2 in a million.

Most diseases have multiple causes, so it is convenient to have a way to disentangle these and express the risk due to a particular cause. This introduces the concepts of risk difference and attributable risk, which indicate the number of cases of a disease among exposed individuals that can be attributed to that exposure (assuming it is a causal influence and not a confounder):

Attributable risk = Incidence in the exposed group − Incidence in the unexposed

This tells us how much extra disease has been caused by this exposure, in absolute terms: 1 case per million persons in the example above. In the case of a factor that protects against disease, such as a vaccination, it tells us how many cases can be avoided.

This idea of attributable risk can also be expressed in relative terms, as a proportion of the incidence in exposed persons, yielding the exposed attributable fraction, EAF (also called the attributable proportion):

EAF = [Incidence _(exposed) – Incidence _(unexposed) ] / Incidence _(exposed)

This statistic can be useful in counselling a patient exposed to a risk factor: “Not only are you at high risk of developing lung cancer, but 90% of your risk is due to your smoking. Quitting could have a major benefit”.

In developing health policies, we can also apply the idea of attributable risk to describing the impact of risk factors on the population as a whole. This introduces measures of population attributable risk (PAR) and of population attributable fraction (PAF): statistics that evaluate the impact of a causal factor by substituting incidence in the whole population for incidence in the exposed (see Nerd’s corner).

Number Needed to Treat

A useful extension of the attributable risk is the concept of “Number Needed to Treat” (NNT). This metric summarizes the effectiveness of a therapy or a preventive measure in achieving a desired outcome. The number needed to treat captures the realities that no treatment works infallibly, and also that some patients recover spontaneously. The NNT estimates the number of patients with a condition who must follow a treatment regimen over a specified time to achieve the desired outcome for one person. The NNT is calculated as the reciprocal of the absolute improvement produced by the treatment. Imagine a medication that cures 35% of the people who take it, while 20% of those who do not take it recover spontaneously. Here, the absolute improvement due to the treatment is 15%. The reciprocal of this is 1 / 0.15, or about 7. So, on average, you would need to treat 7 people to achieve 1 cure from the medication (within the specified time). The NNT can also be applied in describing the value of a preventive measure in avoiding an undesirable outcome. It can likewise be used in calculating the hazard of treatments such as adverse drug reactions, when it is termed “number needed to harm”; this shows the average number of people treated with the drug that would generate one adverse event.

Inferential Statistics

Medical research applies information gained from particular study samples to broader populations, for example to estimate the average birthweight of babies in Canada from records of birthweights in your hospital. This population value is called a “parameter”. The uncertainty of estimating a population parameter from a sample introduces inferential statistics.

Sampling

To provide an accurate estimate of a parameter, a sample should evidently be representative of the population; a random sample offers a good approach. But because people vary, different samples drawn randomly from the same population are likely to give slightly different results purely due to chance variation in who was selected. Random sampling from a population only guarantees that, on average, the results from successive samples will reflect the true population parameter, but the results of any particular sample may differ from those of the parent population, and sometimes substantially. The unintended differences that arise between study findings simply because of their different samples are variously known as “sampling error”, “sampling variation” or “chance error”. But at least we can estimate the likelihood of error in extrapolating, or generalizing, results from a sample to the broader population using inferential statistics such as p-values and confidence intervals (see Additional Material: Statistics).

Estimating a parameter

The confidence interval (or CI) is a statistic used to indicate the probable extent of error in a parameter estimate derived from a sample. An expression such as “mean systolic blood pressure was 120 mmHg (95% CI: 114, 126 mmHg)” indicates that the mean value in the study sample was 120 mmHg, but owing to possible sampling error the actual value in the broader population may not be precisely 120. Based on the size of the sample used and the variability of BP readings within it, there is a 95% probability that the population mean actually lies somewhere between 114 and 126 mmHg. The confidence interval can be represented graphically as a line or error-bar, as seen in Figure 5.8.

Like mean values, odds ratios and relative risks are also reported with confidence intervals. The confidence interval around an odds ratio indicates the likely range within which the true value lies, and also shows whether or not the association is statistically significant. For example, a relative risk of 1.4 (95% CI 0.8, 2.1) means that we can be 95% confident that the true relative risk lies somewhere in the (very wide) range of 0.8 to 2.1: not a lot of confidence! Furthermore, because 1.0 falls within this range, it is quite possible that there is no association in the population.

Significance of differences

Consider an RCT that compares mean blood pressures of hypertensive patients who have been randomly allocated to either a treatment (or intervention) group or a control group. The study hypothesis predicts a difference in BP between the groups, due to the treatment. But as a physician, you are not so much interested in this particular study result, but in how closely it may represent the situation in a broader population, including patients in your practice. Evidently, if the study finding was a chance anomaly in this particular sample, you would not base your practice on those results! We need to compare two possibilities: the study hypothesis of a true difference, and the alternative, “null hypothesis” that the study result was an anomaly and there is no BP difference between such groups in the broader population.

Biostatistical tests help us to evaluate these options. Intuitively, the larger the study sample and the larger the difference it found in mean blood pressures, the more confident we would be that the difference would hold true if the study were to be repeated on another, equivalent sample, such as your patients. The researcher must first choose a probability threshold or p-value that will be used to distinguish between results (here the contrast in mean blood pressure) that may be a chance finding in this particular sample, versus a difference that will be considered real or “statistically significant”. The threshold chosen is usually p < 0.05, or 5% – arbitrary but commonly used. [Now, take a deep breath for the next bit …] A p < .05 here means that the prior probability of getting a contrast in BP as great as the observed result (or even greater) is less than 5%, if the null hypothesis is actually true (i.e., the treatment has no real effect on BP in the population of interest). When a statistical analysis shows a p value less than .05, the difference would be considered statistically significant, and the researcher would generally support the study hypothesis of there being a true difference, while awaiting subsequent studies to replicate this result (see there Here Be Dragons, and the Significant statistical limitations boxes). The actual formula used to calculate a p-value depends on elements of the study design; guidance will be found in a textbook of biostatistics, or perhaps from your colleague with an MSc in epidemiology.

The limits to inferential statistics

A crucial point to recognize is that inferential statistics suggest the level of confidence in generalizing from a random sample of a given size to the population from which it was drawn. But for evidence-based medicine, we frequently wish to generalize to other populations, sometimes in other countries, as illustrated in Figure 5.5. Assessing the validity of this more distant extrapolation requires additional insights into the comparability of the populations and the nature of the topic under study. This information cannot be provided by statistics, yet it forms a critical consideration in evidence-based medicine. Judging the applicability of research results to your patient(s) requires your own knowledge of the similarity or difference between the study sample and your own practice (more on this further on). This introduces the topics of EXTERNAL VALIDITY and sampling bias. The following sections introduce critical appraisal skills for detecting biases in information.

Sources of Error in Studies

Bias

Bias, or the systematic deviation of results or inferences from the truth, is a danger in the design of any study.⁴ Special care is taken by researchers to avoid (or, failing that, to control for) numerous types of bias that have been identified.¹¹ The many possible biases may be grouped into two broad categories: sampling biases (derived from the way that persons in the study were selected) and measurement biases (due to errors in the way that exposures or outcomes were measured).

Sampling (or selection) bias

Simple random sampling aims to select a truly representative sample of people from a broader population; a more formal definition of the idea is that everyone in the population has an equal (and non-zero) chance of being selected. This is especially important in descriptive studies such as those that estimate prevalence. It may be less important in analytic studies that seek to test a theoretical prediction.¹² For instance, a researcher who wants to study the association between obesity and arthritis might be justified in drawing her sample from a population at high risk of obesity in order to get adequate numbers of obese and very obese people in her study.

For practical reasons, very few studies are able to sample randomly from the entire target population, so the researcher usually defines a narrower “sampling frame”, which is assumed to be similar to the entire population. The researcher then draws a sample from that frame. For example, Dr. Rao might sample patients attending Weenigo General Hospital in order to make inferences about patients attending similar hospitals. Sampling bias may then occur at two stages: first, in the choice of sampling frame, because patients attending Weenigo General may differ from patients attending other hospitals in the region and, second, in the method used to draw the sample of patients attending the hospital.

Sampling bias chiefly arises when samples are not drawn randomly. For example, a newspaper advertisement that reads “Wanted: Participants for a study of blood pressure” might attract retired or unemployed people who have the time to volunteer, especially those who have a personal interest in the topic (perhaps they have a family history of hypertension). If these characteristics are, in turn, associated with blood pressure level, an estimate of the population mean BP based on this sample will be biased. Much research is undertaken in teaching hospitals, but patients seen in these centres differ systematically from patients with the same diagnosis seen in rural hospitals—they tend to be sicker, to have more co-morbidities, and often conventional therapy has failed, leading to their referral to tertiary care centres. This can lead to these studies yielding different findings than would be seen in the population of all people with a disease. This is a specific form of selection bias known as referral bias.

Non-response bias

Even if the sampling method is unbiased, not everyone selected will actually participate. If particular types of people do not participate, this can bias the study results. One way to estimate the likelihood of a non-response bias is to compare characteristics of participants, such as their age, sex and where they live, with those of people who refused to participate. However, even if these characteristics match, it does not rule out bias on other characteristics that were not recorded. Those who chose not to respond likely differ in attitudes from those who did, and you will not know how they would have responded. In critically appraising an article you should review the efforts made by the investigator to reduce non-response and judge its likely impact on results.

Information bias: systematic measurement errors

Measurement error refers to deviations of recorded values on a measurement from the true values for individuals in the study. As with sampling error, measurement errors may be random or systematic. For example, social desirability bias refers to systematic response errors whereby people tend to answer questions in a way that will be viewed favourably by others (such as their doctor). Most people report that they are more physically active than the average person, which is illogical. Men tend to exaggerate their height and under-estimate their weight.¹³ Other measurement biases arise from flaws in the questionnaire design: for example, asking people about their physical activity in certain months only may give a biased estimate of their yearly activity level because of seasonal variations in physical activity.  Recall bias commonly occurs in surveys and especially in case-control studies: people’s memories often err. For example, on questionnaires significantly more women report having had a mammography within the past two years than is shown in mammography clinic records.

Bigger, but no less biased

Increasing the sample size or taking more measurements can minimize random sampling and measurement errors but will have no effect on systematic errors; a biased sample or measurement will remain biased no matter how many subjects participate. A large, biased study may be more misleading than a small one!

In the figure, the + sign represents the unknown parameter we are trying to estimate; each red dot represents an estimate of the parameter obtained from a sample (it can equally represent a data point from a repeated measurement). The upper sections of the figure illustrate the presence of systematic error and the sample estimates are off-target or biased. In the presence of systematic error, increasing the sample size or making additional (biased) measurements will not bring the study results closer to the truth; it may simply mislead you into believing that they are more accurate. Increasing the sample sizes or the numbers of estimates in the lower section, where there is little systematic error, will reduce the uncertainty of the estimates.

The patterns in the figure above can also be useful in understanding test and measurement validity and reliability, as discussed in Chapter 6. For this, substitute the word “validity” for “systematic error” and “reliability” for “random error.”

Information Bias: the Objectivity of the Researcher

Whether looking at print or Internet information, you should try to find out as much as possible about the source, to check its credibility and to identify possible conflicts of interest. Trials published by people with a financial stake in the product under investigation are more likely to conclude in favour of the product than are trials published by people with no financial interest. The U.S. Food and Drug Agency and the Federal Trade Commission have proposed questions to ask when judging information sources:

1. Who is behind it?
2. Why is the information being published?
3. Where does the information on the website come from?
4. Are the claims well documented?
5. Who is responsible for the information?
6. How old is the information?

This section has mentioned only a few of the many potential types of study bias. Several authors have undertaken more extensive reviews, as shown in the Nerd’s corner box.

Confounding

As noted above, the possibility of confounding forms a major challenge in deciding whether a statistical association represents a true relationship – especially important in interpreting causal relationships. As an example, a study in the 1960s reported a significant tendency for Down syndrome to be more common in fourth-born or higher order children.¹⁴ There was no obvious sampling or measurement bias and the result was statistically significant, so would you believe it? Your answer may be “yes” in terms of the existence of an association, but “no” if the implication is a causal one. In other words, birth order may be a risk marker, but not a risk (or causal) factor.

Confounding arises when a third variable (or fourth or fifth variable, etc.) in a causal web is associated with both an exposure and the outcome being studied, as shown in Figure 5.7. If this third variable is not taken into account in the study analysis, conclusions about the relationship between exposure and outcome may be misinterpreted. In the Down syndrome example, the mother’s age is a confounding factor in that fourth-born and higher order infants tend to be born to older mothers, and advancing maternal age is an independent risk factor for Down syndrome. In most scientific articles, the first table compares the study groups (e.g., mothers with a Down infant and others without) on a range of variables that could affect the outcome, such as mean maternal age at the child’s birth. This allows the reader to determine whether any of these variables differs between the study groups, and so act as a potential confounding factor that should be adjusted in the analysis.

The Hierarchy of Evidence

Because some study designs provide more reliable evidence than others, the idea of a hierarchy of evidence was proposed in 1979. This arose as a by-product of the Task Force on the Periodic Health Examination that made recommendations on routine screening and preventive interventions. The hierarchy has been modified over the years¹⁶ and here is a generic version, starting with the highest quality of evidence:

Systematic reviews

A common source of bias in summarizing the literature is the omission of some studies from consideration. These could be studies undertaken in countries outside one’s own, or studies published in less well-known journals. While such omissions may simplify the task of summarizing the literature, it can also lead to bias, often by omitting studies that provide discordant views. Systematic reviews aim to identify all relevant studies related to a given treatment or intervention, to evaluate their quality, and to summarize all of the findings. A key feature is the comprehensiveness of the literature review; conclusions should be based on the whole literature, often including the “grey” literature of reports published as working documents or internal reports.¹⁸ A systematic review must follow a rigorous and explicit search procedure that can be replicated by others. The author of a systematic review formulates a narrative summary of the combined study findings (as in the Cochrane Reviews, see below, or in reviews by UpToDate); but where the articles reviewed are similar enough, their data may be pooled into a combined meta-analysis.

Meta-analyses

A meta-analysis provides a statistical synthesis of data from separate yet comparable studies. It is generally accepted that a meta-analysis of several randomized controlled trials offers better evidence than a single trial. The analysis can either pool data from each person in the various studies and re-analyze the combined dataset, or else aggregate the published results from each study, producing an overall, combined numerical estimate of effect. This is normally weighted according to the relative sizes of the studies, and sometimes also by a judgment of their quality. Meta-analyses can be applied to any study design, but if design differences mean that data cannot be pooled, results of different studies may be summarized in a narrative review or presented in a forest plot, as shown in Figure 5.8.

The Cochrane Collaboration

Systematic reviews and meta-analyses are normally undertaken by specialized content and research experts; they often work in teams such as those assembled through the Cochrane Collaboration. This international organization helps scientists, physicians, and policy makers make well-informed decisions by coordinating systematic reviews of the effects of health care interventions. The reviews are published electronically in the Cochrane Database of Systematic Reviews. An early example of a systematic review was that on the use of corticosteroids for mothers in premature labour to accelerate fetal lung maturation and prevent neonatal respiratory distress syndrome. Babies born very prematurely are at high risk of respiratory distress owing to their immature lungs, a significant cause of morbidity and mortality. The results of 21 randomised trials give no indication that the corticosteroid treatment increased risk to the mother, but it produced a 30% reduction in neonatal death and similar benefits on a range of other outcomes. Therefore antenatal corticosteroid therapy was widely adopted to accelerate fetal lung maturation in women at risk of preterm birth.

Meta-analyses are now considered to provide the highest level of evidence, so this has changed the original Canadian Task Force hierarchy of evidence. With the idea of literature review in mind, level I of the hierarchy of evidence shown above is now sub-divided into

1.1 Cochrane reviews
1.2 Systematic reviews
1.3 Evidence based-guidelines
1.4 Evidence summaries.

The Final Step: Applying Study Results to your Patients

The topic of systematic reviews brings us full circle, back to critical appraisal of the literature and evidence-based medicine. Judging whether a study produced results that should guide your care begins with formulating an overall judgment of the quality of the study or studies, and for this there are several checklists. The original ones were developed at McMaster University and published in a series of papers in the Journal of the American Medical Association in 1993 and 1994. These described critical appraisal in judging evidence for causation, prognosis, the accuracy of diagnosis, and effectiveness of therapy. To illustrate the general format, we list some questions used to appraise an article that evaluated the effectiveness of a therapy.

Checklist for study quality

Study objectives

• Was the study question clearly stated? For example, did it follow the PICO format?
• Was the exposure clearly defined? Was the outcome clearly defined? Or were the study objectives and outcomes vaguely worded, as in “To describe the health effects of fast-food consumption”?

Are the results valid?

• Was a suitable study design used? Were the patients randomized?
• Were there systematic errors (biases) in the study execution? For example, was randomization concealed? Were patients aware of their group allocation? (Were the patients “blinded”?)
• If a case-control study, was there a possible misclassification of patient outcomes?
• Were clinicians who treated patients aware of the group allocation?
• Were outcome assessors aware of group allocation? (Were the assessors blinded?)
• Were patients in the treatment and control groups similar with respect to known prognostic variables? (For instance, were there similar numbers of smokers in each group in a study of asthma therapy?)
• Was follow-up complete?
• Were patients analyzed in the groups to which they were allocated?

What are the results?

• How large was the treatment effect?
• How precise was the estimate of the treatment effect?

How can I apply the results to patient care in general?

• Were the study patients similar to patients in my practice?
• Are the likely treatment benefits worth the potential harm and costs?

What do I do with this patient?

• What are the likely outcomes in this case?
• Is this treatment what this patient wants?
• Is the treatment available here?
• Is the patient willing and able to have the treatment?

Once you are satisfied that a study provides a valid answer to a relevant clinical question, check that the results are applicable to your patient population.

Target population

Is the study population similar to your practice population, so that you can apply the findings to your own practice (illustrated in Figure 5.5)? Consider whether the gender, age group, ethnic group, life circumstances, and resources used in the study are similar to those in your practice population. For example, a study of the management of heart disease might draw a sample from a specialist cardiovascular clinic. Your patient in a family medicine centre is likely to have less severe disease than those attending the specialist clinic and, therefore, may respond differently to treatment. Narrow inclusion and broad exclusion criteria in the study may mean that very few of your patients are comparable to the patients studied. Furthermore, the ancillary care and other resources available in the specialist centre are also likely to be very different from those in a primary care setting. If these are an important part of management, their absence may erase the benefits of the treatment under study. Other aspects of the environment may also be different: the results of a study set in a large city that found exercise counselling effective may not apply to your patients in a rural area where there are fewer opportunities for conveniently integrating exercise into everyday life.

Intervention

Is the intervention feasible in your practice setting? Do you have the expertise, training, and resources to carry out the intervention yourself? Can you to refer your patient somewhere else, where the expertise and resources are available? In many cases, practical problems of this type mean that an intervention that has good efficacy in a trial does not prove as effective when implemented in usual practice. The enthusiasm and expertise of the pioneers account for some of this difference; the extra funds and resources used in research projects may also have an effect.

How much does it cost?

The cost includes the money needed to pay for the intervention itself, the time of the personnel needed to carry it out, the suffering it causes the patient, the money the patient will have to pay to get the intervention, ancillary medical services, transport, and time off work. An intervention that costs a great deal of time, money or suffering may not be acceptable in practice.

Intervention in the control group

What, if anything, was done for people in the control group? If they received nothing, including no attention by the researchers, could a placebo effect have accounted for part of the effect observed in the intervention group? In general, new interventions should be compared to standard treatment in the control group so the added benefits and costs of the new can be compared to those of the old treatment.

Your patient’s preferences

Finally, all management plans should respect patients’ preferences and abilities. The clinician should explain the risks and benefits of all treatments—especially novel ones—in terms the patient understands, and support the patient in choosing courses of action to reduce harm and maximize benefit.

The Limits to Evidence-Based Medicine

The central purpose of undertaking research studies in medicine is to guide practice. A fundamental challenge lies in the variability of human populations, so that a study undertaken at a different time and in a different place may or may not provide information relevant to treating the patient in front of you. This is the problem of induction, of generalization when the invariance assumption does not hold. This dilemma demands that the physician should neither unquestioningly apply research results to patient care, nor ignore the evidence of research findings in the rapidly evolving field of medicine (see the Nerd’s Corner box).

Self-test questions

References

1. National Institutes of Health. Citations added to MEDLINE by fiscal year U.S. National Library of Medicine2017 [cited 2017, November]. Available from: https://www.nlm.nih.gov/bsd/stats/cit_added.html.
2. Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005;294(2):218-28.
3. Lett J. A field guide to critical thinking. 1990 [cited 2017, November]. Available from: https://www.csicop.org/si/show/field_guide_to_critical_thinking.
4. Porta M, editor. A dictionary of epidemiology. New York (NY): Oxford University Press; 2008.
5. Sackett DL, et al. Evidence-based medicine – how to practice and teach EBM. London: Churchill-Livingstone; 2000.
6. Schwandt TA. Qualitative inquiry: a dictionary of terms. Thousand Oaks (CA): Sage Publications; 1997.
7. Nkwi P, Nyamongo I, Ryan G. Field research into socio-cultural issues: methodological guidelines. Yaounde, Cameroon: International Center for Applied Social Sciences, Research and Training/UNFPA, 2001.
8. Richards I, Morese JM. Read me first for a users’ guide to qualitative methods. Thousand Oaks (CA): Sage Publications; 2007.
9. Cockburn J, Hill D, Irwig L, De Luise T, Turnbull D, Schofield P. Development and validation of an instrument to measure satisfaction of participants at breast cancer screening programmes. Eur J Cancer. 1991;27(7):827-31.
10. Smith GCS, Pell JP. Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ. 2003;327(7429):1459-91.
11. Sackett DL. Bias in analytic research. J Chron Dis. 1979;32:51.
12. Miettinen OS. Theoretical epidemiology: principles of occurrence research in medicine. New York (NY): John Wiley; 1985.
13. Connor-Gorber S, Shields M, Tremblay MS, McDowell I. The feasibility of establishing correction factors to adjust self-reported estimates of obesity. Health Reports. 2008;19(3):71-82.
14. Renkonen KO, Donner M. Mongoloids: their mothers and sibships. Ann Med Exp Biol Fenn. 1964;42:139-44.
15. Anderson GL, Judd HL, Kaunitz AM, et al. Effects of estrogen plus progestin on gynecologic cancers and associated diagnostic procedures: The Women’s Health Initiative randomized trial. JAMA. 2003;290:1739-48.
16. Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nursing. 2003;12(1):77-84.
17. GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ. 2004;328(7454):1490.
18. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151:W65-W94.
19. Tonnelli MR. Integrating evidence into clinical practice: an alternative to evidence-based approaches. Journal of Evaluation in Clinical Practice. 2006;12(3):248-56.
20. Solomon M. Just a paradigm: evidence-based medicine in epistemological context. European Journal of Philosophical Science. 2011;1:451-66.
21. Worrall J. Evidence in medicine and Evidence-Based Medicine. Philosophy Compass. 2007;2(6):981-1022.

Français (French)