Evaluation of Canada’s Action Plan Against Racism

2. Methodology

The evaluation followed the scope and methodology set out in an evaluation plan developed during a planning phase completed by GCS prior to commencement of the evaluation. The evaluation planning phase was undertaken between December 2009 and January 2010 and was completed in consultation with the CIC evaluation team and CAPAR program representatives, including members of the IWG. The evaluation plan was designed to align with the Treasury Board Policy on Evaluation (April, 2009).

The following section outlines the evaluation issues and questions, data collection methods, and methodological considerations for the evaluation.

2.1. Evaluation issues and questions

The horizontal evaluation of CAPAR examined issues related to relevance, design and delivery, and performance. Table 2-1 details the evaluation issues and questions addressed in the evaluation (see Appendix B for the complete evaluation matrix, which also includes specific indicators and methodologies for each evaluation question).

2.2. Data collection methods

The evaluation of CAPAR included the use of multiple lines of evidence and complementary research methods to help ensure the reliability of information and data collected. The following data collection methods were used to gather data for the evaluation:

• key stakeholder interviews;
• document review;
• literature review; and
• synthesis of individually funded program evaluation findings.

Each of these methods is described in more detail in the following sections.

2.2.1. Key stakeholder interviews

A total of 26 interviews were completed for the evaluation. Interviews were undertaken with five key stakeholder groups, as described below.

CAPAR Secretariat: Interviews were conducted with representatives of the Citizenship and Multiculturalism Branch who were involved with the CAPAR Secretariat. This group was interviewed primarily to gather information on relevance (i.e., alignment of CAPAR objectives and departmental priorities) and the coordination of CAPAR.

Senior Management from the Citizenship and Multiculturalism Branch: Interviews were conducted with senior management from the Citizenship and Multiculturalism Branch. The primary purpose of these interviews was to gather information on program delivery and performance.

CAPAR-funded program representatives: Program representatives from the funded initiatives were interviewed, including those that were not implemented or were cancelled. Information obtained from this group was intended to inform the evaluation team on the relevance, delivery and outcomes of their respective initiatives, as well as on the overall success of CAPAR as a horizontal initiative.

Multiculturalism Champions: Interviews were conducted with Multiculturalism Champions from non-funded initiatives. The primary purpose of these interviews was to gather information on anti-racism activities being undertaken in the non-funded departments ^{Footnote 4}.

Other stakeholders: Other stakeholders interviewed included members of non-governmental organizations (NGOs) and a crown corporation, working in the field of anti-racism. A list of stakeholders was developed in conjunction with CIC’s Citizenship and Multiculturalism Branch. This interview group was expected to provide input on the need for, and utility of, a federal initiative against racism.

The number of interviews completed for each interview group is illustrated in Table 2-2 (see Appendix C for a list of interviewees).

Interviews were conducted both in-person and via telephone. Different interview guides were developed for each of the interview groups and the interview questions were aligned with the evaluation questions identified in the evaluation matrix (see Appendix D for the interview guides). All interviewees received a copy of the interview guide in advance of the interview. The results of the interviews were summarized in an interview notes template and were then coded and analyzed to determine key themes.

2.2.2. Document review

Documentation was reviewed primarily to inform an assessment of the relevance and design and delivery of the Initiative. The following types of documentation were reviewed during the evaluation:

Foundation documents: including the Treasury Board submission and the Memorandum to Cabinet for CAPAR.

Corporate / accountability documents: including CIC’s Departmental Performance Reports, Reports on Plans and Priorities, the RMAF, and other documentation that provided information on CIC and GoC priorities (e.g., Speech from the Throne).

Acts, charters, international conventions: Canada has an extensive legal and policy framework and many international obligations related to multiculturalism, rights and freedoms, and protection from discrimination. These legislative documents and international conventions were included in the review.

Reviews: CAPAR has undergone a number of reviews, including a baseline assessment, an evaluability assessment, and a management review. All of these reviews were included in the document review.

Program materials: including presentations, reporting templates and tools, consultation material and results.

The document review also provided information to assess whether CAPAR was an efficient use of resources. Financial information from CAPAR was analyzed to examine budgeted and actual costs of CAPAR. The document review was conducted using a customized template that was organized according to evaluation questions and indicators. Annex E contains a list of documents that were reviewed for the evaluation.

2.2.3. Literature review

GCS conducted a literature review of horizontal management, which examined: the various rationales for horizontal initiatives; the various types/classifications (i.e., informal and formal); and success factors for management

. Using the information from the literature review, GCS established criteria for assessing the effectiveness of CAPAR as a horizontal initiative (e.g., rationale for horizontality, effectiveness of management). See Appendix F for a list of the assessment components.

2.2.4. Synthesis of evaluation findings

A major source of information for the evaluation was the evaluations undertaken by the individual programs funded under CAPAR. GCS reviewed a total of six

evaluation reports and extracted relevant information as per the indicators in the CAPAR evaluation matrix. The information available from these program evaluations was used to assess the overall performance of the CAPAR initiative. To a lesser extent, the evaluations also provided information with respect to relevance and program design/delivery.

2.3. Limitations and considerations

A few methodological limitations and considerations for the evaluation should be noted.

Difficulty in assessing the overall impact of CAPAR as a whole

The wide scope of CAPAR activities, program beneficiaries, and target audiences, resulted in a range of program outputs and outcomes. Common outputs and outcomes could only be articulated at a very high level for the Initiative. This lack of definition hindered the ability of the evaluation to measure and aggregate program impacts for CAPAR as a whole. Furthermore, the evaluation did not attempt to examine changes in levels of racism in Canada over time.

Limited information on CAPAR non-funded Initiatives

Limited research was conducted to determine the scope of non-funded initiatives being undertaken by other departments. Interviews with Multicultural Champions were intended to inform the evaluation team of anti-racism activities being undertaken in other government departments and agencies; however respondents had varying degrees of familiarity with CAPAR and with anti-racism activities in their respective departments.

Amount and types of information available on the different CAPAR-funded initiatives varied considerably

The evaluation relied heavily on individual funded program evaluations to assess the results of CAPAR. In examining the documents available on the different CAPAR-funded initiatives, the evaluation team found considerable differences between evaluation reports in terms of the amount of information and the program results. Many evaluations on activities and outputs rather than program outcomes. In regards to initiatives that were cancelled or not implemented, the evaluation team found there was limited information available (i.e., ARTCI and III).

• Previous
• 1
• 2
• 3
• 4
• 5
• 6 (current)
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• Next