Chapter 7 Applications of Research Methods

Applications of Research Methods in Public Health Surveillance and Programme Evaluation

After reading this chapter, you will know how to:

1.  Access and collect health information to describe the health of a population:
• Explain the importance of disease surveillance in maintaining population health
• Explain different approaches to surveillance:
Passive surveillance
Active surveillance
• Cite potential sources of data for surveillance:
Hospital information
Vital statistics
Sentinel surveillance

2. Describe the role of physicians in reporting and responding to disease;

3. Know the defining characteristics of an outbreak and  how to recognize one when it occurs from an epidemic curve:
Point source
Continuous source
Intermittent exposure
• Index case with limited spread
Propagated spread

4. List different types of health services research:
• Community health needs assessment
Economic evaluation
Programme evaluation.  

Linking these topics to the Medical Council exam objectives, sections 78-2, 78-3 and 78-4.

Writing a discharge summary

Dr. Li, medical internist at Weenigo Hospital, is writing up Catherine Richards’s discharge notes. Mrs. Richards had a worsening cough and was admitted from her nursing home with signs of broncho pneumonia and mild cardiac failure secondary to the pneumonia. Initial work-up also showed that Mrs. Richards’s diabetes was out of control and blood cultures grew pneumococcus. She responded well to therapy and went back to the nursing home a week after admission. Dr. Li is tired and is wondering what she should enter in the box marked “principal diagnosis” and just how much what she writes really matters – does anyone do anything with this information? She wonders if she should make reference to Mrs. Richards’s stroke, which caused her immobility that is likely to have contributed to the development of broncho pneumonia. She would like to put “metabolic syndrome” as one of the underlying causes for the admission, but the last time she used the term in a discharge summary the archivist warned her that there was no code for it. She is also wondering if she should report the invasive pneumococcal infection to the public health department.


Evidence-based planning for any health service or preventive programme requires information on the types and distribution of health problems in the population. This is the task of surveillance, which refers to the systematic, ongoing collection and analysis of population-level health information in order to guide the design of public health and preventive interventions.


Surveillance is the systematic collection, analysis and timely dissemination of information on population health to those who need to know, so that action can be taken. Surveillance provides information about patterns of health and disease, and changes in these patterns. This information can guide prevention and control efforts as well as contribute to planning health services and, subsequently, to evaluating their impact. Because health can be influenced by personal, community and environmental factors, surveillance data come from various sources, including:

  • Vital statistics data, such as births and deaths
  • Environmental data on air and water quality
  • Health services indicators, such as hospital discharges
  • Census information on the population, such as income, language, and ethnic group.

Much of the routine work of regional and provincial public health authorities as well as the Public Health Agency of Canada involves surveillance.

Role of clinicians in public health surveillance

While public health services take overall responsibility for coordinating health surveillance, individual clinicians play a major role by routinely reporting occurrences of particular contagious diseases that they see. It is primary care physicians who typically see the initial cases in what may become an epidemic. They may see more people than usual presenting with a particular condition, and patients might mention other people with the same symptoms. The primary care physician must inform the public health service. The public health authorities assemble such case reports to gain a broad perspective on the extent and severity of a health threat, and to initiate prevention and control measures such as contact tracing, as required. As well as completing mandatory notification of reportable diseases, physicians also contribute to surveillance systems by completing death certificates and by entering accurate diagnoses on hospital discharge forms and billing forms.

Types of Surveillance

There are two main types of surveillance. Long-term, passive surveillance monitors general trends in health status and health determinants. This has documented the current epidemic of obesity and changes in trends of certain cancers, for example. Active or short-term surveillance searches for emergent diseases or outbreaks such as SARS or pandemic influenza A (H1N1), and helps society respond rapidly to new threats.

Passive surveillance

The “passive” in passive surveillance refers to the role of the agency responsible for surveillance: they wait for reports to come to them. Reports may take the form of routinely collected data, such as hospital discharge summaries, mortality data, or physician billing data. Reports also include reports of notifiable diseases that must, by law, be reported.

Notifiable diseases are those considered to be of public health importance. Legislation requires physicians and laboratories to report them to the local public health agencies once they are suspected or diagnosed. This allows public health to identify possible outbreaks early and implement prevention and control measures in a timely manner. However, health professionals do not always realize the importance of the information they provide, so under-reporting can be a problem. In addition, some provinces have legislation that requires notification of possible outbreaks, even if the disease itself is not notifiable. For example, the Quebec Public Health Law demands “any physician who suspects the presence of a threat to the health of the population must notify the appropriate public health director.” Further information on notifiable disease is available in Chapter 11.

Notifiable diseases

While the concept of notifiable (or reportable) disease applies mainly to communicable diseases that pose a threat of epidemic spread, it also applies to certain non-communicable conditions. In Quebec, poisoning with certain agents such a heavy metals or carbon monoxide is notifiable, as are certain diseases caused by non-transmissible environmental agents.

The designation of a disease as notifiable may change, reflecting the level of threat it may pose to the community. As new infectious diseases emerge they may be added to the list, especially if they are commonly transmitted by international travel. A number of contagious conditions are reportable to the WHO, such as plague, cholera, yellow fever and other diseases identified in the International Health Regulations. The WHO also monitors world trends in poliomyelitis, malaria, SARS, and influenza A. Their Global Outbreak Alert and Response network is an international collaboration of experts to provide rapid response to outbreaks of international importance. Canadian federal law only provides a framework for reporting notifiable diseases, although the Public Health Agency of Canada requires the reporting of some diseases at a national level. Notification is mandated by provincial legislation and the list of diseases varies from province to province. You can consult Notifiable Diseases Online to see which diseases are notifiable and how frequently they occur.

Hospital and billing data

As a resource for passive surveillance, hospital discharge summaries can provide useful information on patterns of disease and on the therapies being used. However, because availability of services greatly influences their use, comparisons of these data between places and over time are of limited value for disease surveillance. Similarly, physician billing data can be used, but new methods of physician remuneration and inaccurate and missing diagnoses limit the usefulness of this data source.

Vital statistics: births and deaths

Recording births and deaths is mandatory in most countries and provides basic vital statistics. Physicians are responsible for entering the causes of death on the death notification. Causes are coded according to the International Classification of Diseases (ICD) so countries can compare death rates and trends in disease patterns. Again, the accuracy of this information depends on the accuracy with which the original reporting physician recorded the causes of death because these are rarely confirmed by autopsy.

The ICD coding rules require the reporting physician to distinguish between the underlying cause of death and the precipitating cause or causes. The underlying cause is “the disease, injury or pathological condition that initiates the chain of events leading to death.”1 Data compilers follow a protocol that governs the decision about what disease to record as the main cause of death. For example, a patient with lung cancer who dies of pneumonia is likely to have his death attributed to lung cancer even if the pneumonia was unrelated to the cancer. Death records are routinely used to compile national information on trends in diseases such as cancer.

Diagnostic coding

Dr. Li had been unsure of how to record the principal diagnosis for Catherine Richards. She looks in the ICD to see whether it lists metabolic syndrome and the 10th edition does not. However, it does list obesity under code E66, so she uses that instead. She wonders if there will be a revision to the ICD soon. She notices from the web that the WHO is inviting collaborators on developing the 11th edition, due in 2018.

Dr. Li also asks her secretary, Mrs. Bibi, to check about reporting the pneumococcal disease. After contacting the local public health department, her secretary comes back with the news that invasive pneumococcal infection is notifiable. Mind you, says Mrs. Bibi, she had been chatting to Roxanne, a secretary in the infectious disease laboratory and Roxanne said that the laboratory automatically passes on information about a number of diseases including pneumococcal infection to the public health unit.

ICD History

The International Classification of Disease originated in 1891 when the International Epidemiological Society started to classify causes of death. By 1900, 26 countries had adopted the classification. Since then, most countries have adopted it and the classification has been revised to reflect the development of medical science. After 1946, the WHO took over responsibility for coordinating revisions to the list. The tenth revision was published in 1994 and adopted in Canada for coding causes of death in 2001.

Active surveillance

Active surveillance means that those responsible for it play an active role in data gathering. This is consequentially resource intensive and is usually undertaken for a specific purpose. For example, the Canadian Paediatric Society routinely sends letters to every paediatrician asking them to report on cases of rare conditions, such as acute flaccid paralysis, to assess the success of polio vaccination. It also asks for information on cerebral oedema in diabetic ketoacidosis, which allowed the condition to be characterized and guidelines for its management to be developed. The Society then reports the data to the Public Health Agency of Canada.

Health surveys

Surveys, such as the Canadian Community Health Survey (CCHS) and the national census can also be viewed as active surveillance. The CCHS was initiated in 2001. Every two years it gathers data on general health and health habits from a large random population sample. In the intervening years, it collects data on specific health topics from a smaller sample. Surveys can also target particular groups, such as injection drug users or people with a particular diagnosis, to document changes in patterns of behaviour that may affect disease or transmission. This is termed “second-generation surveillance”. The WHO gives the example of regular recording of information on HIV risk behaviours, using this to warn of or explain changes in levels of infection.

The census

Information on the population denominators required for interpreting most surveillance information comes from the census. The first national Canadian decennial census was carried out in 1871. In keeping with international norms, there has been a census every 10 years since then, always in the years ending in 1. Since 1956 there has been an additional census in the years ending in 6. Both censuses cover the entire population and collect basic demographic data—about eight questions. In addition, more detailed information is collected from a random 20% sample of the population, covering a range of demographic, social and economic topics (about 50 questions), but not including health.

Sentinel surveillance

The notion of sentinel refers to clinicians keeping watch for particular diseases of interest. Selected clinicians gather data and pass them on to those responsible for the surveillance. For example, the Canadian Primary Care Sentinel Surveillance Network links selected family health teams via an electronic record surveillance system. This can be used both to report rare events (such as side-effects they observe from influenza immunizations) and to help improve the quality of care. If the sample of physicians is carefully designed, estimates can be made of the population incidence of the event of interest without the need to survey the entire population.

Watching influenza

FluWatch is Canada’s national surveillance system for influenza. It provides a national picture of influenza activity throughout the year to health professionals and the public.


  • Offers early detection of influenza in Canada
  • Provides up-to-date information on influenza activity in Canada and abroad to professionals and the public
  • Monitors circulating strains of influenza virus, including new sub-types and antiviral resistance
  • Contributes virological surveillance information to the WHO to guide the development of vaccines for following flu season.

FluWatch information comes from several sources:

  • Laboratory reports of influenza and other respiratory viruses collected by sentinel laboratories across Canada
  • Strain identification and antiviral resistance for circulating influenza viruses from the National Microbiology Laboratory
  • Primary care consultations for influenza-like illnesses (ILI) from sentinel practitioners across Canada
  • Regional influenza activity levels as assigned by provincial and territorial FluWatch representatives
  • Paediatric hospital admissions and mortality associated with influenza through the Immunization Monitoring Program ACTive (IMPACT).

More information is available from the Public Health Agency web site.

Monitoring surveillance

When surveillance shows changes in patterns of a disease, we should ask:

  1. Is the change real?

If a disease is rare, chance fluctuations can cause large proportionate variations. So, could an apparent rapid increase have happened because the initial number of cases was small? Grouping the data across times or places can sometimes resolve this problem; however, different groupings can sometimes lead to different interpretations of the data.

  1. Could the change be due to changed reporting practices? Are there variations in the precision of the data?

Did something change reporting procedures? Screening programmes may increase the apparent prevalence of a disease because they identify cases that were previously undiagnosed. For example, the recent increase in reported Chlamydia cases may result in part from new routine screening, especially since the advent of non-invasive urine testing (see L.K. Altman, Sex diseases still rising; Chlamydia is leader, New York Times November 14, 2007).

  1. Might the change be due to a shift in definition of the disease?

Are the same diagnostic criteria being used? For example, changes in disease definitions from the ninth to tenth version of the International Classification of Disease in 2001 led to a drop in the number of deaths attributed to falls and pneumonia, and a rise in the number attributed to cancer. Similarly, changes in the diagnostic criteria of autism led to mistaken reports of its increasing incidence.

Patterns of Disease Development in a Population: the Epidemic Curve

When our efforts to prevent disease fail and an outbreak develops, the resulting distribution of cases over time may take various forms that are called epidemic curves. These trace the evolution of a population outbreak, equivalent to the idea of the natural history of a disease for an individual case. The natural history of a population outbreak is most evident in infectious disease, but also occurs in situations such as a chemical spill that causes respiratory disease or, on a much longer time-scale, in non-communicable, chronic diseases. The shape of the resulting epidemic curve can be used to propose hypotheses on the nature of the disease and its mode of transmission. As well as showing the temporal pattern, the curve shows the magnitude of the outbreak (the number of cases), the likely incubation period for the condition, and can reveal outliers (in time and perhaps in place).

To characterize different types of outbreak, the Centers for Disease Control classify epidemic curves based on the suspected type of exposure.2 In a common source outbreak people are exposed to a single noxious influence. The source may occur for a brief time or it may persist, depending on the circumstances. When the exposure is very brief, most people get sick at one incubation period following exposure, and this is called a point source outbreak—an example would be Staphylococcus aureus food intoxication from tainted food at a wedding dinner (awkward for the hosts). This produces a single curve that wanes quickly, as long as there is no person-to-person spread (Figure 7.1).

figure7.1 - Chapter 7 Applications of Research Methods
Figure 7.1: Point source exposure epidemic curve


The distribution of cases over time reflects the differential incubation period for different individuals. Perhaps some ate more of the tainted food and became ill sooner than others, or they were more susceptible; perhaps some took longer to seek care and so be included in the data collection.

Continuous source: Sometimes a common source exposure can be prolonged, as with exposure to a contaminated water supply, or via a restaurant’s faulty refrigeration system that does not get fixed. Here, cases arise over an extended period but still originate from a common or single source. The resulting epidemic curve becomes longer and flatter, indicating the longer duration of the source and the variation in incubation periods between people: see Figure 7.2. The curve ends when the source of the contamination is corrected or when all susceptible people develop immunity. The relative flatness of the curve suggests that the infection comes from a common source and that there is no person-to-person spread; otherwise the number of cases would grow over time as one person infects others.

figure7.2 - Chapter 7 Applications of Research Methods
Figure 7.2: Continuous source epidemic curve


Intermittent exposure: Figure 7.3 shows an irregular pattern of cases that reflects the timing and extent of repeated exposures. It is not immediately clear whether this is a common source, such as an industrial contaminant emitted at intervals, or whether it arises from several sources, such as a series of outbreaks of food poisoning occurring at different summer camps for children. The gaps between the outbreaks could initially suggest person-to-person transmission followed by an incubation period, but the successive peaks do not become larger and merge as they would if the outbreaks were due to infectious spread, with one person infecting several. Hence, the epidemic curve in Figure 7.3 suggests a non-transmissible condition.

figure7.3 - Chapter 7 Applications of Research Methods
Figure 7.3: Intermittent outbreak epidemic curve


Index case with limited spread. Person to person spread is illustrated in Figure 7.4, which shows the typical pattern arising when a single index case (for example, a traveller returning from abroad) infects other people with an incubation period. This is called a point source with secondary transmission. The outbreak wanes when the infected people no longer transmit the infection to other susceptible people, perhaps because of successful control measures (isolation or quarantine).

figure7.4 - Chapter 7 Applications of Research Methods
Figure 7.4. Secondary transmission from a single index case


Propagated Spread. This begins like an infection from an index case (Figure 7.4), but the secondary cases of the disease then act as sources to infect new people who, in turn, serve as sources for yet other cases. Figure 7.5 illustrates how this produces successively taller peaks in each generation (i.e., secondary and tertiary cases). Peaks are initially separated by one incubation period, but then tend to merge into one large peak and the epidemic continues until the remaining pool of susceptible individuals shrinks or until control measures take effect. This pattern occurs with diseases such as measles that spread from person to person.

figure7.5 - Chapter 7 Applications of Research Methods
Figure 7.5: Index case with propagated spread epidemic curve

Health Services Research

The rising expense of medical innovations, plus the additional demands for care the innovations create, mean that publicly funded systems have difficulty paying for every new treatment that becomes available. Finite budgets mean that managers and governments have to make choices about which programmes to fund. These decisions consider whether the programmes are required, whether they are beneficial, and whether they may be improved upon. Research that informs such decisions is known as Health Services Research, which can be divided into needs assessment, economic evaluation, and overall programme evaluation.

Needs assessment

A key question in planning health services concerns how many and what types of health services the population needs. A first approximation is to base an estimate of needs on current utilization data and historical patterns of care, taking demographic trends into account. An advantage of this approach is that utilization data are usually readily available and can be compared to other jurisdictions in order to estimate the appropriate level of service provision. However, utilization data generally reflect historical supply and demand for services (and these two influence each other) much more than they reflect the more abstract notion of need.3, 4

An alternative approach distinguished between need and demand, and brings the ideas of prevalence and of the effectiveness of care into the calculation: it argues that we have no need for care that is ineffective.5-7 This approach starts from the prevalence of a condition in the whole population, not just those who consult. It then incorporates evidence of the effectiveness of available interventions (preventive as well as curative) to estimate the numbers of required interventions. These estimates can be compared with the current level of service provision to identify procedures that may be over- or under-supplied. An example of assessing the need for stroke services is presented in the Illustration Box.

Defining need

Need for care can sometimes be defined in absolute terms: a person is injured and needs emergent care to save his life. But the difficulty is to know when to stop; dying patients could be considered to need heroic interventions but these may prolong the condition without curing it. Concern over inappropriate intervention has led to defining need in relative terms, considering the benefit that would accrue from any care that could be provided.

Using the relative approach, Acheson defined need as “the ability of an individual to benefit from health care, and [need] exists where he or she has a condition for which there is an effective and acceptable intervention.”5 In this conception, a patient may demand treatment (e.g. an antibiotic for a viral infection) but if the requested treatment will be ineffective he or she has no need for it. While this can appear counterintuitive, from a health services perspective it sensibly points out that we should direct expenditures of scarce resources towards activities that provide benefit, and not waste resources on ineffective treatments.

A more recent synthesis between the absolute and relative perspectives holds that need and demand should not lie in conflict, but that patient and doctor should negotiate expectations for care. The service providers only provide services, and the patient and family must play their part in determining what services are to be received.

The distinction between need and demand also reminds us of the factors affecting the definition of disease (see Chapter 1). This is influenced both by demand pull (requests for treatment) and by the supply push (e.g., pharmaceutical companies marketing new products whose sale helps to fund research to develop new products to meet future demands).

How many stroke beds?

A study estimating the requirement for stroke services in Eastern Ontario was undertaken in five steps:

  1. Identify stroke risk factors through literature reviews and consultations with experts
  2. Estimate the frequency of strokes and of the identified risk factors in the region
  3. Identify effective health services targeting each condition or risk factor from systematic reviews and published practice guidelines
  4. From this, determine the number and type of health services required
  5. Compare these estimates with the actual provision of services.

The study showed several mismatches between the estimated need and actual provision of stroke services. The approach offers an effective way of making health care planning decisions.8

Economic evaluation

Limited resources mean that choices must be made of which services to fund, and health economics uses information on the costs and outcomes of care to guide these choices. Health economics is a branch of economics that considers costs, benefits, resource allocation, use, inputs, outputs and outcomes of all forms of health care.1 Costs are the value of resources used to deliver a service. These will include items such as staff wages, buildings, equipment, maintenance, and supplies. Costs are usually measured in terms of monetary value. Outcomes or benefits represent the results of a treatment or intervention, and can include symptom relief, survival rates, or improvement in quality of life, as reviewed in Chapter 6. In economic evaluations, outcomes are often measured in terms of QALYs or DALYs (QUALITY-ADJUSTED LIFE YEARS and DISABILITY-ADJUSTED LIFE YEARS).

The key economic assumption is that any health service should deliver the greatest possible benefit per unit of cost. There are four main types of economic evaluation, which are similar in the way they assess cost inputs but differ in the way they assess outputs.

  1. Cost minimization analysis. This is the simplest approach, applicable when the benefits of two interventions are the same, so favouring the cheaper option. An example would be the choice between a name brand and a generic drug.
  2. Cost benefit analysis. This assesses benefit in terms of monetary value (dollars). An example would be a company’s fitness programme that has a cost per participant, but whose benefit to the company can be costed in terms of reduced sick leave and increased productivity.
  3. Cost effectiveness analysis. This assesses output in terms of improved health outcomes, such as symptom control or survival. An example is a study that compared the standard method of in vitro fertilization with a novel one. The study considered the different costs associated with the two approaches and used the number of pregnancies resulting in live births as the outcome.9
  4. Cost-utility analysis. This is a variant of cost-effectiveness, where the measure of outcome is adjusted to include a judgment of utility, via QALYs or DALYs or the health-adjusted life expectancy (HALE). The major advantage is that this approach allows for comparisons across different procedures and their related outcomes. Note that many published papers purporting to be cost-effectiveness analyses are actually cost-utility analyses.

Programme evaluation

A health programme is a planned series of activities; the plan indicates the objective, financing, roles and responsibilities, and intended outcomes of the programme. The national breast cancer screening programme, cardiac rehabilitation programmes and the annual flu vaccination drive in Dr. Rao’s clinic are all examples of health programmes.

Because of the emphasis on the efficiency of services, evaluation has to be built into all programmes to demonstrate that public funds are being spent responsibly. Programme evaluation is the “systematic gathering, analysis, and reporting of data about a programme to assist in decision making”. It should answer the question “Does the programme achieve its stated goals and objectives?” (Goals give a broad statement of what a programme hopes to achieve, while objectives are more precise targets that form steps on the path to achieving the overall goal). The procedures of quantitative and qualitative research described in Chapter 5 apply to programme evaluation, using the outcome measures described in Chapter 6.

Programmes can be evaluated in terms of their structure, process, and outcome, as proposed by Donabedian in 1966:10

  • Structure: the adequacy of facilities, equipment, staffing, and/or administration of the programme. (Should Dr. Rao hire another nurse to help Nurse Jennings on their influenza immunization drive?)
  • Process: what the programme does. For example, in a hospital this could cover the adequacy of history taking, physical examination, diagnosis, therapy, or tertiary prevention. From a public health perspective, process might cover the number of screening tests performed, the number of radio spots aired, etc. (How well did Dr. Rao organize his clinic on vaccination night? Was there enough vaccine?)
  • Outcome: the results or outputs of the programme. Patient attitudes and satisfaction, their level of rehabilitation or residual disability, or the death rates observed among them. (What proportion of the patients in the target group was immunized on vaccination night; will they come again next year; how many patients got influenza this year compared to last year?)

Possible patient outcomes of health services

Fletcher et al. offered a slightly different version of the five Ds that were introduced in Chapter 1, and these are often used as a framework for measuring patient outcomes in health services research:11

Dissatisfaction Emotional reaction to disease and its care, including sadness or anger.
Disease Symptoms, physical signs, and laboratory abnormalities.
Discomfort Symptoms of pain, nausea, dyspnoea, itching, and tinnitus.
Disability Impaired ability to undertake usual activities at home, work, or recreation.
Death A poor outcome, if it occurs prematurely or if it is painful.

There are many guidelines on how to evaluate a health programme; one example is given in the Nerd’s corner box.

Steps in evaluating a programme

Stage 1: The Logic Model.

The first step is to work from a “logic model” for the programme, which should have been developed during the planning process for a complex programme. The logic model outlines how an intervention (such as a public health or prevention programme) is expected to produce results. The purpose is to ensure that all people involved are aware of how their activities contribute to the overall goal: it gives the big picture. It also helps guide evaluation of the programme by identifying measurable intermediate goals that track progress toward the overall goal and thereby identify where failures occurred if the end goal is not met. The elements in a typical model include a description of the situation or problem to be addressed, the inputs or resources required, the activities to be undertaken, the outputs, and the anticipated outcomes.

Here is an example of an imaginary logic model for the dental health programme described in “an application of the Ottawa Charter” in Chapter 4.12

    • Situation: Evidence shows the oral health of children in Glasgow to be among the worst in Western Europe
    • Inputs: A collaboration between the dentists, primary care physicians and city authorities
    • Activities: Creating healthy public policy; Supportive environments; Developing personal skills; Strengthening community action; Reorienting health services (see The Ottawa Charter for Health Promotion in Chapter 4).
    • Outputs: numbers of nursery schools contacted; numbers of children receiving dental care; improvements in dental health indicators
    • Target systems: nursery schools, families with children under 5; community centres
    • Outcomes:
      • Short term: participation by mothers in oral health training programmes; reduced consumption of caries-inducing foods; changes in foods served in nursery schools
      • Medium-term: improved oral health profile
      • Longer-term: reduction in the percentage of adolescents and adults with dental problems.

Stage 2: The Evaluation

The Public Health Agency of Canada lists five key steps in evaluating a programme. Again, these may be applied to the dental health promotion programme.12

PHAC Evaluation Guide Illustration, as applied to dental health promotion programme
1.   Identify the purpose of the evaluation:
  • State the purpose of the evaluation
  • Develop a logic model for the programme, indicating how the structure and process are expected to achieve the outcome
  • List the stakeholders
  • Develop evaluation questions and check the feasibility of undertaking an objective evaluation


“To assess the dental health outcomes of an oral health promotion programme by secondary analyses of routine caries datasets for Glasgow 5-year-olds over the interval from 1997-98 to 2003-04.”12
The logic model is shown above.Stakeholders included the City of Glasgow Health department; University of Glasgow Dental Hospital; Scottish Office Department of Health; several community groups.The feasibility of accessing dental records was confirmed
2.   Select appropriate methods:
  • State expectations
  • Formulate a data collection plan
  • Develop a logistics plan and run a feasibility check


A self-report survey appeared infeasible; there was need for expert assessment of oral health to record outcomes.

“In Glasgow, cross-sectional caries surveys of randomly selected 5-year-old children in primary school reception classes are carried out routinely as part of a national programme.”

Permission was granted to retrieve anonymous data from previous surveys.

3.   Develop tools:
  • Review existing measures
  • Select questions and response categories
  • Plan a quality assessment for the data collection



The surveys “are conducted according to the British Association for the Study of Community Dentistry standardized criteria”

“…and involve annual national training and calibration exercises immediately preceding each survey…”

4.   Gather and analyze data:
  • Data collection and pre-test
  • Quantitative and/or qualitative analysis


(This study used secondary data from an existing data set)

An index of area socioeconomic deprivation was added to the oral health data, linked by postcode.

Quantitative analyses used changes in mean oral health scores over time, comparing experimental city districts to districts in which the programme was not implemented.

5.   Make decisions:
  • Interpretation of results
  • Formulate an action plan
  • Produce a report.
“This paper provides evidence of positive and reproducible outcomes following targeted community-based oral health promotion activities in SES-challenged communities.”

“The earlier in infant life that caries risk factors are ameliorated, the greater the impact on caries incidence for the individual child and caries prevalence at the community level.”12

Dr. Li’s diagnosis revisited

  1. Dr. Li was wondering about the importance of documenting diagnoses in hospital discharge summaries…

Public health practitioners use hospital discharge information and billing codes to monitor trends in incidence, prevalence and clinical severity of various acute and chronic diseases. Dr. Li’s hospital data provides valuable information for program planning and evaluation and enables international comparisons of disease rates.

  1. How are diseases classified internationally?

Through the ICD-10 classification (International Classification of Disease, 10th revision).

  1. Which types of infections need to be reported to the public health department? Why is this important?

While the concept of notifiable (or reportable) disease applies mainly to communicable diseases that pose a threat of epidemic spread, it is also applies to certain non-communicable conditions. Reporting is important as front-line clinicians are generally the first to spot early cases in what may become an epidemic.

Self-Test Questions

1. Identify the differences between active, passive and sentinel surveillance, and provide an example of each.

Passive surveillance involves reviewing routinely collected information (sometimes collected for other purposes, such as physician billing records) to record trends in disease. The information is then passed on to public health authorities so that action may be taken if required.  An example would be routine reporting of cases of notifiable diseases.Active surveillance is often used under more urgent circumstances, and implies setting up a system for collecting data on a disease specifically in order to track its development. This is often initiated by public health authorities who are concerned about a particular public health threat.  An example would be reporting cases of H1N1 influenza during a suspected outbreak.

Sentinel surveillance is a combination of both: it involves establishing a carefully selected network of physicians or public health units who routinely report information on diseases of interest. An example would be a network of primary care physicians who are chosen according to where they practice in order to study and contrast patterns of sexually transmitted infections in rural and urban areas.

2. A systematic review has concluded that a new programme to reduce hypertension is extremely effective. In your capacity as advisor to the Minister of Health, how would you determine the need for such a programme and how would you evaluate its impact in terms of outcome and cost?

Turning an effective programme into a population-wide policy implies presenting a political argument that would explain the justification for diverting funds away from another programme (opportunity cost).An example of the steps to be followed to formulate an effective political and economic argument for a new hypertension policy are as follows:

First, you would have to demonstrate the current costs of hypertension (in terms, for example, of numbers of people affected, or their subsequent morbidity and mortality, the costs of treating conditions that arise secondary to hypertension, etc).

Second, summarize the current approach to managing hypertension: how well does the current system work; how badly broken is it and how much of an advantage would the new programme be?

Next, you would have to assess the feasibility of implementing the proposed new programme on a population-wide basis (do you have the facilities, staff, equipment for implementing the programme; could it be implemented in all areas, or would it create disparities in access?)

Finally, you need to anticipate the costs of implementing the programme, and compare these with the benefits that would accrue: will an early intervention for hypertension actually reduce costs of subsequent treatment? Ideally, this cost-benefit analysis should be compared to the cost-benefit of existing ways of managing hypertension and of any other programmes that could be abandoned to release funds for this new programme.

Many of these steps would involve modelling the programme impact and costs, based on estimation procedures that rely on previous surveys (for example, covering the prevalence and geographic distribution of hypertension).


  1. Last JM. A dictionary of public health. New York: Oxford University Press; 2007.
  2. Centers for Disease Control. Interpreting an epidemic curve. In: Epidemiology in the classroom: steps in an outbreak investigation [cited 2016 January]. Available from:
  3. Morgan M, Mays NHWW. Can hospital use be a measure of need for health care? J Epidemiol Comm Health. 1987;41(4):269-74.
  4. Labelle R, Stoddart G, Rice T. A re-examination of the meaning and importance of supplier-induced demand. J Health Econ. 1994;13(3):347-68.
  5. Acheson RM. The definition and identification of need for health care. J Epidemiol Community Health. 1978;32:10-5.
  6. Stevens ARJ. Introduction. In: Stevens A, Raftery J, editors. Health care needs assessment: the epidemiologically based needs assessment reviews. Volume 1. Oxford: Radcliffe Medical Press; 1994.
  7. Frankel SJ. Health needs, healthcare requirements, and the myth of infinite demand. Lancet. 1991;337:1588-90.
  8. Hunter DJW, Grant HJ, Purdue MPH, Spasoff RA, Dorland JL, Bains N. An epidemiologically-based needs assessment for stroke services. Chronic Diseases in Canada. 2004;25(3/4):138-45.
  9. Polinder S, Heijnen EMEW, Macklon NS, Habbema JDF, Fauser BJCM, Eijkemans MJC. Cost-effectiveness of a mild compared with a standard strategy for IVF: a randomized comparison using cumulative term live birth as the primary endpoint. Human Reproduction. 2008;23(2):316-23.
  10. Donabedian A. Evaluating the quality of medical care. Milbank Quarterly. 1966;83(4):691-729.
  11. Fletcher R, Fletcher S. Clinical epidemiology: the essentials. Baltimore (MD): Lippincott, Williams & Wilkins; 2005.
  12. Blair Y, Macpherson L, McCall D, McMahon A. Dental health of 5-year olds following community-based oral health promotion in Glasgow, UK. Int J Paediatric Dentistry. 2006;16:388-98.
printfriendly pdf email button notext - Chapter 7 Applications of Research Methods

Share and Enjoy !

0 0

Français (French)