Section 1: Evaluation of Family Violence Initiative Activities at the Public Health Agency – Introduction

1. Introduction

1.1 Purpose and scope of the evaluation

Senior management requested this evaluation to support program planning and decision-making. Its purpose is to explore the current relevance and performance of the long-established Family Violence Initiative activities at the Public Health Agency and, as appropriate, make recommendations to improve the design and delivery of the program.

The evaluation focuses on the Family Violence Initiative activities of the Family Violence Prevention Unit (Centre for Health Promotion, Health Promotion and Chronic Disease Prevention Branch) between 2004 and 2011.

Evaluation issues and questions address both relevance (continued need for this program and alignment with federal government roles and priorities) and performance (achievement of expected outcomes and demonstration of efficiency and economy) of the Family Violence Initiative activities of the Public Health Agency.

Relevance:

1. Do the Public Health Agency Family Violence Initiative activities continue to address a demonstrable need?

2. Are the Public Health Agency Family Violence Initiative activities responsive to the needs of Canadians?

3. Do the Public Health Agency Family Violence Initiative activities remain a priority for the federal government?

4. What is the federal public health role in addressing family violence?

5. What are the links between the Public Health Agency Family Violence Initiative activities and the Public Health Agency's role in public health?

6. In what ways is the Public Health Agency best positioned to contribute to preventing family violence in Canada?

7. What are the objectives of the Family Violence Initiative activities at the Public Health Agency? What has been accomplished through the Family Violence Initiative activities at the Public Health Agency?

8. What are the priorities and the vision for the Family Violence Initiative activities at the Public Health Agency moving forward? Do we need to refocus the Public Health Agency's family violence prevention activities?

Performance:

9. Were the Family Violence Initiative activities at the Public Health Agency managed effectively to facilitate the achievement of its stated role?

10. Have these activities leveraged stakeholder relationships to contribute to evidence-based knowledge?

This is not an evaluation of the broader horizontal federal Family Violence Initiative. It does not examine the role and activities of the other 14 departments that participate in the federal Family Violence Initiative or other Public Health Agency-led activities outside of the Centre for Health Promotion's Family Violence Prevention Unit. Therefore it does not examine whether or not the federal Family Violence Initiative overall should continue and, if it should continue, which department should lead it. The assumption made at the outset of this evaluation was that PHAC will continue as the lead department in the immediate future.

In addition, this evaluation does not explore either the immediate or longer term impacts of the Family Violence Initiative activities of the Public Health Agency. There is no performance measurement strategy (no logic model, no defined program outcomes) for this program. Due to the time sensitive nature of this senior management requirement to support decision-making, the focus of this expedited evaluation was the relevance and performance of current Family Violence Initiative activities at the Public Health Agency. Performance data, while limited, were analyzed and incorporated as part of the performance story for this report.

This report provides a description of the Family Violence Initiative activities at the Public Health Agency, and their history and current context. It presents an assessment of links, outlines challenges and opportunities, and culminates in seven findings which in turn lead to three recommendations. Appendices A and B provide additional detail about related international activities.

Management and staff should use the findings and recommendations of this evaluation for program planning and decision making.

1.2 Methodology

Approach and design

This evaluation was conducted by Evaluation Services, a group internal to the Public Health Agency but not involved in the program area responsible for the administration of the Family Violence Initiative activities.

The overall approach to this evaluation integrated two conceptual models: goals-based and process-based evaluation.

  • A goals-based model allowed the evaluators to explore whether the Family Violence Initiative activities at the Public Health Agency are meeting the predetermined goals and objectives set out in its original mandate. Changes in the approach to the prevention of family violence in Canada over the past 30 years required a review of the relevance of these activities.
  • A process-based model was appropriate for the other purpose of the evaluation, which was to fully understand how Family Violence Initiative activities at the Public Health Agency perform today. Although this long-standing initiative has evolved since it was first established in the late 1980s, this evaluation provided an opportunity to explore improvements to the delivery of its mandate at the Public Health Agency.

Given the short time frame for completion of the evaluation, Evaluation Services used a non-experimental cross-sectional design to take a retrospective look at this long-established program. Short time lines were driven by senior management decision making requirements. Since little baseline information was available, this design allowed evaluators to analyze information from one point in time from multiple sources. It explored how the Family Violence Initiative activities at the Public Health Agency align with and add value to the Public Health Agency's roles and priorities, as well as how the various components of these activities have performed.

Senior management requested that this evaluation review the relevance of the Family Violence Initiative and the performance of its activities. The challenges for the evaluation were:

  • a paucity of current performance measurement data available for the initiative
  • the long history of the initiative (almost 30 years) and minimal evaluative analysis undertaken over that time
  • a shift in the Canadian context for family violence since the establishment of the initiative.

In addition to the perspectives of a broad cross-section of Public Health Agency program management and staff, the evaluation considered the perspectives of some of the major stakeholders; in particular, the perspectives of a selection of other government departments were considered.

Methods

This evaluation used multiple lines of evidence, including literature, document reviews and face-to-face interviews. It was important to triangulate document and literature reviews with the interviews of relevant stakeholders. For an international perspective, the evaluators also reviewed the approaches for addressing family violence in three other countries (Appendix B).

Evaluation tools, such as interview guides, are available upon request.

1. Literature review

Evaluators conducted a streamlined review of literature to explore the nature of family violence and family violence prevention in Canada and internationally. In particular, the literature review addressed evaluation issues related to relevance (evaluation questions #1, #2 and #4). The purpose of this review was to obtain information about whether the Family Violence Initiative represents a legitimate and necessary role for the Government of Canada and, if so, to determine best practices which could improve the design and delivery of the initiative. The evaluators also looked for information on the connection between family violence and the social determinants of health

The 49 documents reviewed included literature suggested by program staff (including unpublished literature) and documents identified through internet searches. As well, the evaluators conducted bibliographic key word searches on English and French academic research published in the last 10 years; in particular, literature about family violence prevention and the role of public health. The following databases were searched: Scopus (which contains the content of Medline & Embase), CINAHL (Cumulative Index to Nursing and Allied Health Literature) and Global Health.

The evaluators conducted reviews of three other countries' approaches to the leadership and coordination of family violence prevention and dissemination of information. These reviews addressed the evaluation issues related to relevance (evaluation question #4) and performance (evaluation questions #8 and #9).

The other countries reviewed were:

  • United States: Centers for Disease Control and Prevention (CDC), Division of Violence Prevention
  • Australia: Australian Domestic and Family Violence Clearinghouse
  • New Zealand: Family Violence Prevention Strategy and Ministry of Social Development.

This purposeful sampling strategy focused on anticipated international best practices in developed countries. The evaluators compiled and synthesized a review of available internet information and documentation available from each organization.

2. Document review

Evaluation Services reviewed documentation available about the Family Violence Initiative activities at the Public Health Agency. The staff of the Family Violence Initiative at the Public Health Agency identified many of the documents. This review addressed the evaluation issues related to relevance (evaluation question #3, #5 and #6) and performance (evaluation questions #7, #8, #9 and #10). In total, evaluators reviewed 130 documents.

The types of documents reviewed included:

  • previous performance reports and evaluations of the federal Family Violence Initiative
  • terms of reference and records of decision from various internal and external committee meetings
  • correspondence and communication related to the Family Violence Initiative activities at the Public Health Agency
  • statistics on the dissemination of information through the National Clearinghouse on Family Violence
  • reviews of other federal horizontal initiatives.
3. Interviews with Public Health Agency key senior managers and staff (current and former), experts in other federal government departments, and an international expert in the field of family violence prevention

The purpose of the interviews was to obtain a further description of how various elements of the program have been implemented, and to receive input on the relevance of the Family Violence Initiative activities at the Public Health Agency and whether or not it achieved expected outcomes. The interviews addressed particular evaluation issues related to relevance (evaluation question #5 and #6) and performance (evaluation questions #7, #8, #9 and #10).

Evaluators conducted the interviews using structured interview guides that were developed in discussion with internal and external experts in the field of family violence prevention. Evaluators intended to audio record and transcribe the interviews, but time and budget constraints did not allow the interviews to be transcribed.

Managers and staff of the Family Violence Initiative at the Public Health Agency identified key informants. The sampling approach was purposive. Evaluation Services notified key informants of the evaluation and requested their participation. From April 11 to July 22, 2011, 15 one-hour interviews were conducted. 21 interviewees participated in total: several interviews involved more than one key informant. Each key informant addressed one or more of the evaluation questions.

The breakdown of interviewees by type of key informant was as follows:
Interview sub-group Number of interviewees
Public Health Agency senior managers and staff 13
Experts in other federal government departments 7
International expert in the field of family violence prevention 1
TOTAL 21

Limitations

Most evaluations are confronted with constraints that may have implications for the validity and reliability of evaluation findings, conclusions and recommendations. This section discusses the limitations with respect to the design and methods for this particular evaluation. Also discussed are the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Limited breadth and depth of analysis

With a short time frame (April to July 2011) for completion of data collection for this evaluation, due to a management requirement for decision-making, there was a limited breath and depth of analysis of some data, such as limited: in-depth financial review, provincial and territorial perspectives, and the relationship with the activities of the other departments that participate in the federal Family Violence Initiative. However, the interviews with departmental and external key informants provided insight into each of these areas. The time frame also limited the number of international and domestic comparators reviewed. In particular, the international reviews were not intended to be a representative sample of jurisdictions; rather, they were a purposeful sample selected for their anticipated best practices.

Reliability and validity issues with data sources

Due to the large number of potential informants (internal and external), the evaluators relied on the recommendations of staff of the Family Violence Initiative at the Public Health Agency. Some of these informants worked with the Family Violence Initiative staff and may have had a vested interest in the Family Violence Initiative.

Several validity issues are related to the qualitative data sources. The interviews were in part retrospective and their accuracy may be questioned. Secondly, detailed notes were taken during interviews, but operational constraints prevented the interviews from being transcribed verbatim. Thirdly, staff turnover at the Family Violence Initiative meant there was limited knowledge of the history of the initiative's activities (although interviews with former staff compensated for the lack of corporate knowledge).

Limited quantitative performance data

Limited quantitative performance data were available. Due to the variances in the systems for electronic data tracking of Clearinghouse resources, dissemination of resources was not consistently tracked. Evaluators were able to review some Clearinghouse information on dissemination patterns for family violence resources, but little other program performance data were available for analysis, such as data on the use or uptake of the information available through the Clearinghouse.

The evaluation team attempted to mitigate the imbalance between qualitative and quantitative data by incorporating the multiple lines of evidence presented above (i.e. the literature review, the document review and the key informant interviews) into its approach.

Page details

Date modified: