Evaluation of the Citizenship Awareness Program

2. Methodology

2.1. Evaluation scope and approach

The evaluation approach and methodology were set out in an evaluation plan developed for both the Citizenship Awareness and Citizenship Acquisition, Confirmation and Revocation components of the Citizenship Program. The plan was developed between February and June 2012 and completed in consultation with all CIC Branches involved in their delivery. Although planning encompassed both components, each will be evaluated separately in order to ensure robust coverage of the different aspects of the program. A Terms of Reference for the Citizenship Awareness evaluation was approved by CIC's Departmental Evaluation Committee in July 2012.

The evaluation examines the Citizenship Awareness program activity (PA3.2.1 of the CIC Program ment Architecture (PAA)), including the Grant to the Institute for Canadian Citizenship (ICC). The Grant to the ICC was included in this evaluation as its objectives are ed with those of the Citizenship Awareness program. As well, combining the two elements into one evaluation was deemed to be an efficient means of clustering evaluations in order to meet requirements within existing resource levels.

The present evaluation focuses on the activities undertaken, outputs produced, and results achieved for Citizenship Awareness over the five-year time period between fiscal years 2007-08 and 2011-12. The formative evaluation of Citizenship Week conducted in 2004Footnote 9 solely on the activities related to Citizenship Week, and therefore provided little information to inform the present evaluation. An audit of the Citizenship ProgramFootnote 10 was conducted by CIC's Internal Audit Branch in 2011, and internal and external reviews were also conducted in 2010-11. However, these reviews primarily explored issues related to the Department's processing of citizenship applications, rather than the promotion of Canadian citizenship.

A program logic approach was used for the study design of the evaluation, supported by a detailed evaluation matrix. The logic model depicts the activities, expected outputs, and expected outcomes for both components the Citizenship Program (see Figure 2-1). The specific expected outcomes addressed in the evaluation of the Citizenship Awareness component are:

  • Newcomers and established Canadians participate in CIC promotional events and/or receive CIC promotional materials;
  • Clients of CIC promotional activities have knowledge of the responsibilities and privileges associated with Canadian citizenship;
  • Eligible applicants become Canadian citizens and persons who are ineligible do not;
  • Canadian citizenship is a valued status;
  • Evidence-based recommendations and/or decisions are made; and
  • Decision-makers and staff are equipped with the skills, knowledge and tools required.

Figure 2–1: CIC Citizenship Program logic model

Figure described below
Text version: CIC Citizenship Program logic model

Figure 2-1 illustrates the logic model for CIC’s Citizenship Program. The logic model provides a visual representation of the Citizenship Program, and contains three main components: Program Management, Awareness, and Acquisition, Confirmation and Revocation. Activities, outputs and expected outcomes are identified for each component. The present evaluation is assessing expected outcomes in relation to the Awareness and Program Management components, which have been delineated by a red dashed line in the logic model. These outcomes are denoted as “as assessed in the present evaluation” in the description provided below.

The program management component of the Citizenship Program has two main activities:

  1. Set direction, plan, monitor and report; and,
  2. Enable program delivery.

The first activity under program management contributes to a group of seven related outputs:

  • Priorities and plans;
  • Research;
  • MCs and TB Subs;
  • Strategic policies;
  • Performance measurement framework;
  • Performance data, reports; and,
  • Evaluations.

These outputs contribute to one immediate outcome: Evidence-based recommendations and/or decisions are made (assessed in the present evaluation).

The second activity under program management contributes to a group of six related outputs:

  • Legislative and regulatory initiatives;
  • Operational policies;
  • Advice, interpretations and recommendations;
  • Procedural and guidance documents;
  • Training; and,
  • Tools and checklists.

These outputs contribute to one immediate outcome: Decision-makers and staff are equipped with the skills, knowledge and tools required (assessed in the present evaluation).

Together, the two immediate outcomes from the program management component contribute to both the awareness component and the acquisition, confirmation and revocation component.

The awareness component of the Citizenship Program has one main activity, which is to promote citizenship.

This activity contributes to a group of five related outputs:

  • Public outreach events & products;
  • Regular, enhanced ceremonies;
  • Reaffirmation ceremonies;
  • Educational/promotional tools and materials; and,
  • Partnerships and linkages.

These five outputs contribute to one immediate outcome: Newcomers and established Canadians participate in CIC promotional events and/or receive CIC promotional materials (assessed in the present evaluation). This immediate outcome links to sub-activity 3.2.1 under the departmental Program ment Architecture.

This immediate outcome contributes to two intermediate outcomes:

  1. Clients of CIC promotional activities have knowledge of the responsibilities and privileges associated with Canadian citizenship (assessed in the present evaluation); and,
  2. Eligible applicants become Canadian citizens and persons who are ineligible do not (assessed in the present evaluation).

Together, the two intermediate outcomes contribute to the ultimate outcome: Canadian citizenship is a valued status (assessed in the present evaluation). This ultimate outcome links to program activity 3.2 under the departmental Program ment Architecture.

The logic model for the Citizenship Program also illustrates activities, outputs and expected outcomes in relation to the Acquisition, Confirmation and Revocation component (not assessed in the present evaluation).

The acquisition, confirmation and revocation component of the Citizenship Program has two main activities:

  1. Process applications (grants, proofs, renunciations and resumptions); and,
  2. Conduct revocations.

The first activity under acquisition, confirmation and revocation contributes to a group of ten related outputs:

  • Applications;
  • QA checks;
  • Immigration, security and criminal checks;
  • Evaluation of language and knowledge;
  • Hearings;
  • Interviews;
  • Decisions, appeals;
  • Ceremonies; and,
  • Grants, proofs, renunciations.

The second activity under acquisition, confirmation and revocation contributes to a group of six related outputs:

  • Tips from CIC, others;
  • Partnerships with law enforcement agencies;
  • Decisions;
  • Letters;
  • Appeals; and,
  • Revocations.

These two groups of outputs contribute to one immediate outcome: Applications and revocations are efficiently and effectively processed. This immediate outcome links to sub-activity 3.2.2 under the departmental Program ment Architecture.

This immediate outcome contributes to all four intermediate outcomes:

  1. Clients of CIC promotional activities have knowledge of the responsibilities and privileges associated with Canadian citizenship; and,
  2. Eligible applicants become Canadian citizens and persons who are ineligible do not;
  3. Eligible applicants obtain proof of citizenship and those who are ineligible do not; and
  4. Persons who obtained Canadian citizenship fraudulently have their citizenship revoked.

Together, these four intermediate outcomes contribute to the ultimate outcome: Canadian citizenship is a valued status.

2.2. Evaluation questions

The evaluation was designed to address two broad themes: relevance and performance. In keeping with the requirements of the Directive on the Evaluation Function (Treasury Board of Canada Secretariat, 2009), program relevance was assessed in terms of: (1) continued need; (2) consistency with respect to federal roles and responsibilities; and (3) ment with government and departmental objectives and priorities. Program performance was assessed by examining program results in terms of: (4) effectiveness; and (5) efficiency and economy (resource utilization). The evaluation questions, organized by evaluation issue, are presented in Table 2-1. The complete evaluation matrix, which includes specific indicators and methodologies for each evaluation question can be found in the Technical Appendices.

Table 2–1: Summary of evaluation issues and questions
Evaluation Issue Evaluation Question Section Reference #Footnote 11
Relevance 1. Is there a continued need to promote the value of citizenship? 3.1.1
2. Are the expected results of the Awareness Program ed with CIC and GoC objectives and priorities? 3.1.2
3. Is the promotion of citizenship a federal responsibility? What other parties have a responsibility to promote citizenship? 3.1.3
Performance 4. Do decision-makers have access to timely, accurate, evidence-based information to make decisions and manage the program? 3.2.5
5. Do staff have the knowledge, tools, and functional guidance required to perform their duties? 3.2.5
6. To what extent do newcomers & established Canadians participate in CIC promotional activities and/or receive CIC promotional materials? 3.2.1
7. To what extent do clients of CIC promotional activities have knowledge of the responsibilities and privileges associated with Canadian citizenship? 3.2.2
8. How did promotional activities influence newcomers' decisions to apply for citizenship? 3.2.3
9. To what extent have promotional activities influenced whether Canadian citizenship is a valued status? 3.2.4
10. To what extent is the Awareness Program delivery model efficient? 3.2.5
11. Have there been unintended impacts of the Awareness program? Throughout Section 3

2.3. Data collection methods

The evaluation included the use of multiple lines of evidence to help ensure the strength of the information and data collected. Program representatives were consulted in order to obtain administrative data and program documents for review, as well as to assist in identifying potential interviewees and locations for site visits.

Following the completion of data collection, each line of evidence was analyzed and several brainstorming sessions were held with project team members to develop the overall evaluation findings, conclusions and recommendations. Each of the lines of evidence is described in more detail below.

2.3.1. Interviews

A total of 67 interviews were completed for the evaluation (Table 2-2). Interviews were undertaken with 5 key stakeholder groups. Individuals were selected in consultation with program representatives based on their knowledge of citizenship awareness activities. The interviews were conducted to respond to all of the questions in the evaluation matrix, covering areas of program relevance and performance.

Table 2–2: Summary of interviews completed
Interview Group Number of Interviews
Citizenship representatives at NHQ 15
Citizenship representatives in Regional/Local offices 29
Citizenship Judges 10
Partners/stakeholders 10
Representatives from other government departments 3
Total 67

Interviews were conducted either in-person or by telephone, and interviewees were provided with a copy of the relevant guide in advance of their interview (see the Technical Appendices for the interview guides).

Where qualitative information is presented in the report, the scale shown in Table 2-3 is used. Note that, in some cases (i.e., where the number of interviewees was too small or where the question yielded more descriptive information), the responses were not coded and a summary approach was used to analyze the information.

Table 2–3: Interview data analysis scale

All
Findings reflect the views and opinions of 100% of the interviewees.
Majority/Most
Findings reflect the views and opinions of at least 75% but less than 100% of interviewees.
Many
Findings reflect the views and opinions of at least 50% but less than 75% of interviewees.
Some
Findings reflect the views and opinions of at least 25% but less than 50% of interviewees.
A few
Findings reflect the views and opinions of at least two respondents but less than 25% of interviewees.

2.3.2. Site visits

Site visits were conducted in January and February 2013 in five communities: Vancouver, Calgary, Mississauga, Montreal and Halifax. The goal of the site visits was to observe the daily operational context of the program at the regional/local level and to better understand the interactions between citizenship processing and promotional activities. The experience also enhanced the evaluation team's understanding of the different perspectives of staff in the regional/local offices compared to those of staff in headquarters with respect to the overall relevance and impact of promotional activities. The following activities were undertaken during the site visits:

  • Tours of the regional/local offices, including on-site testing and citizenship ceremony facilities;
  • Interviews with CIC regional/local managers and staff, citizenship judges, and representatives of partner organizations (e.g., language training centres, settlement service providers and public institutions), to discuss their role in promoting citizenship;Footnote 12 and
  • Observation of citizenship ceremonies, including an assessment against a set of standardized criteria related to facilitation and delivery. A total of nine citizenship ceremonies were observed: two in Vancouver, two in Calgary, two in Mississauga, two in Montreal and one in Halifax.

2.3.3. Ceremony exit surveys

Working with CIC regional/local office staff, a short exit survey was distributed to new citizen participants (18 years of age and older) at the ceremonies attended by evaluation team members during the site visits. The survey (and postage-paid return envelope) was included in their citizenship welcome packageFootnote 13 and the ceremony clerk mentioned the survey in the ceremony instructions.Footnote 14

The survey questionnaire included questions asking participants about the main reasons for their decision to become citizens, the parts of the ceremony that were most important for them, whether anything could have been improved, how the ceremony made them feel, and related impacts of their participation. See the Technical Appendices for additional information on the methodology as well the ceremony exit survey questionnaires.

A total of 217 new citizens completed the ceremony exit survey (59% completed it in English and 41% completed it in French). A slightly modified version of the ceremony exit survey was also created to be distributed at a few ICC ceremonies.Footnote 15 These ceremonies were not attended by members of the evaluation team. A total of 19 new citizens participating in an ICC ceremony completed the exit survey (all in English). The response rate, broken down by ceremony location, is illustrated in Table 2-4 below.

Table 2–4: Exit survey response rate
Ceremony location Number distributed Number of responses Response rate
Vancouver 123 27 22%
Calgary 159 33 21%
Mississauga 220 25 11%
Montreal 657 115 17%
Halifax 75 17 23%
Overall 1,234 217 18%
ICC ceremonies (4 in the Greater Toronto area and 1 in Calgary) 236 19 8%

2.3.4. Survey of new citizens

A survey of new citizens was conducted to answer evaluation questions related to the design and delivery and performance of the program. The survey included questions related to the citizenship experience of new citizens (such as the reasons for their decision to become citizens and how important it was for them), as well as questions about the effectiveness of CIC's citizenship awareness activities and products (mainly the citizenship ceremony and the study guide).

A survey firm was engaged to conduct the survey by telephone with individuals 18 years of age and older who received their citizenship between 2007-08 and 2011-12, using a random sample of 10,000 new citizens, drawn from a population of 688,651 eligible clients in CIC's administrative databases.Footnote 16 In total, 657 new citizens completed the survey, mostly by telephone; 4% of respondents completed the survey online. With a confidence level of 95%, percentages reported for the survey of new citizens have a margin of error of ±3.8%. See the Technical Appendices for a more detailed description of the survey methodology as well as the survey questionnaire.

2.3.5. Administrative data review

The following types of administrative data were reviewed to obtain information on the operations of the program:

  • Operational data from internal reports and the Global Case Management System (GCMS) were used to obtain information on the number of citizenship grants produced and the ceremonies delivered in the years under study;
  • Administrative data from CIC communications function were used to identify the number, type, and cost of citizenship promotional activities conducted by the Communications Branch;
  • Financial information gathered from SAP was analyzed to establish the costs for the program; and
  • Web analytics data were used to assess program reach.

2.3.6. Literature review

A literature review was conducted to examine the evaluation questions related to program need and the role of the federal government. The research included academic and technical journals, publicly available information from foreign government websites, articles by think-tanks and/or non-governmental organizations (e.g. Organization for Economic Cooperation and Development (OECD)), as well as public opinion research. The review was also useful in gaining a better understanding of citizenship policy approaches and their influence on decisions related to citizenship promotional activities. The review also looked at approaches to citizenship promotion in other countries (Australia, New Zealand, United Kingdom, and the United States). See the Technical Appendices for a partial list of the articles and reports reviewed.

2.3.7. Document review

A review of over 50 relevant program documents was conducted to provide background and context to inform an assessment of the relevance and design and delivery of the program. Official government documents such as Speeches from the Throne and budget speeches, as well as policy and strategic documents were reviewed for contextual background and for information on CIC and GoC priorities. Legislation (e.g., the Citizenship Act) and regulations were reviewed to determine responsibility for citizenship promotion. Departmental reference documents including Department Performance Reports (DPR) and the Reports on Plans and Priorities (RPP), along with Ministerial speeches, news releases and internal program documents were used to address specific evaluation questions. See the Technical Appendices for a partial list of the documents reviewed.

2.3.8. Institute for Canadian Citizenship (ICC) case study

A special case study was conducted to examine the efforts and early results related to the ICC as part of the evaluation of the broader Citizenship Awareness Program, as well as to fulfill requirements under the Financial Administration Act for the grant to the ICC. The case study assessed the Grant to the ICC in relation to the objectives set out in the Grant Agreement. Findings from the case study have been integrated throughout the evaluation report, where appropriate, with a view to informing results for the program as a whole. The case study used four main methodologies in its examination of the activities and results of the ICC:

  • Document review;
  • Key informant interviews;
  • A review of administrative data provided by the ICC; and
  • Exit survey results from the ICC ceremony respondents.

A case study report was prepared (see Appendix A).

2.4. Strengths, limitations and considerations

The evaluation contained a balance of qualitative and quantitative lines of evidence and allowed for the triangulation of research findings. The strengths, limitations, their possible impacts on the analysis, and mitigation steps are discussed below.

The evaluation included numerous and diverse lines of evidence and solicited input from a large and varied set of stakeholders. This allowed many stakeholder perspectives to be included and also helps to ensure that findings are reliable. Findings documented by more than one line of evidence demonstrate consistency and thus increase the strength of conclusions.

In addition, site visits to the five different communities allowed for the first-hand observation of program operations in the regions, the delivery of citizenship ceremonies and the participation of new citizens in the citizenship ceremonies. It also allowed for the evaluation team to meet with CIC staff, judges and other stakeholders at the local level with a role in citizenship awareness and promotion, enhancing opportunities for formal and informal connections and the team's overall understanding of program implementation.

While mitigated by the use of multiple lines of evidence, some limitations should be noted:

  • The evaluation focussed primarily on the perspectives of new citizens (and not permanent residents or those born in Canada) despite the fact that the target audience for the Citizenship Awareness Program is all Canadians. Understanding the perspectives of those permanent residents that do not choose to become citizens and those born in Canada would add depth to the findings; however, this was not feasible due to constraints on conducting public opinion research.
  • A change in the Department's financial coding in 2011-12, and incomplete Citizenship Awareness expenditure information in the program budget, resulted in the use of estimates rather than actual figures in the analysis of resource utilization.
  • Different internal reporting formats, as well as an operational change from tracking the number of on/off site ceremonies in favour of tracking the number of standard/enhanced ceremonies, made it difficult to determine the number of new citizenship grants for a particular ceremony type over the period of study.
  • The response rate to the exit survey for those participating in ICC ceremonies was relatively low. Also, distribution of the exit survey was limited to ICC ceremonies held in the Greater Toronto Area (GTA) and Calgary. Therefore, results of the exit survey for those participating in the ICC ceremonies are not considered representative, and should be interpreted with some caution.

These limitations were taken into account when developing the findings, conclusions and recommendations put forward in this evaluation.

Page details

Date modified: