Evaluation of the Atlantic Canada Opportunities Agency Policy, Advocacy and Coordination program activity

 

FINAL REPORT

Evaluation Unit, Audit and Evaluation Directorate

Atlantic Canada Opportunities Agency

March 6, 2012

 

List of Acronyms

A&D - Aerospace and defence

ACE - Atlantic Canada Energy

ACOA - Atlantic Canada Opportunities Agency

APRI - Atlantic Policy Research Initiative

APT - Atlantic Population Table

BDP - Business Development Program

CEAP - Canada’s Economic Action Plan

CAF - Community Adjustment Fund

CD - Community Development

CED-Q - Canada Economic Development for Quebec Regions

DG - Director general

DPR - Departmental Performance Report

EC - Economics and social science services

ECBC - Enterprise Cape Breton Corporation

ED - Enterprise Development

ESC - Evaluation steering committee

ExCom - Executive Committee

FedNor - Federal Economic Development Initiative for Northern Ontario

FTE - Full-time equivalent

G&C - Grants and contributions

HO - Head office

IRB - Industrial and regional benefits

NRCan - Natural Resources Canada

O&M - Operations and maintenance

OBT - Opportunity-based teams

PAA - Program activity architecture

PAC - Policy, Advocacy and Coordination

PMS - Performance measurement strategy

RDA - Regional development agency

RInC - Recreational Infrastructure Canada

SME - Small and medium-sized enterprises

TBS - Treasury Board of Canada Secretariat

VP - Vice-president

WD - Western Economic Diversification Canada

 

Table of Contents

Acknowledgements

Executive Summary

1. Introduction

1.1 Evaluation Overview

1.1.1. Evaluation Rationale

1.1.2. Evaluation Design and Methodology

1.1.3. Evaluation Limitations and Challenges

2. Profile of the Policy, Advocacy and Coordination Program Activity

2.1. Context

2.2. Program Theory

2.3. Program Accountability and Governance

2.4. Expenditure Profile

2.4.1. Grants and Contributions Expenditures

2.4.2. Operations and Maintenance Expenditures

2.5. Salary Expenditures

3. Findings: Relevance

3.1. Legitimate and Necessary Role for Policy, Advocacy and Coordination

3.1.1. Policy, Advocacy and Coordination as Outlined in the ACOA Act and the Program Activity Architecture

3.2. Alignment with ACOA and Government-wide Priorities and Strategies

3.3. Extent to which Stakeholder Needs Are Met

4. Findings: Performance – Effectiveness

4.1. Achievement of Expected Results

4.2. Effectiveness of Governance Structure

4.3. Contribution to Priority Setting and Strategic Direction

4.4. Accessibility and Relevance of Policy Research and Analysis

5. Findings: Performance – Efficiency and Economy

5.1. Mechanisms that Support Efficiency and Economy

5.1.1. Governance, Roles and Responsibilities

5.1.2. Coordination and Communication Mechanisms

5.1.3. Performance Measurement and Reporting

5.2. Evidence of Efficiency and Economy

5.2.1. ACOA Policy Advocacy and Coordination Versus Comparable Organizations

5.3. Best Practices

6. Conclusions and Recommendations

6.1. Relevance

6.2. Performance

Appendix A – Program Activity Architecture Chart

Appendix B – Summary of 2010 Atlantic Policy Research Initiative Evaluation

Appendix C – Evaluation Question Matrix

Appendix D – Data Collection Methodology

Appendix E– Alignment of Policy, Advocacy and Coordination Conclusions and Recommendations

Appendix F – Management Action Plan

Appendix G – Reference Notes

 

List of Figures

Figure 1: Policy, Advocacy and Coordination Logic Model

Figure 2: Policy, Advocacy and Coordination Expenditures by Region (2005-2006 to 2009-2010)

Figure 3: Number of BDP Projects (Planning and Studies) by Region (2005-2006 to 2009-2010)

Figure 4: Number of O&M Projects by Region (2005-2006 to 2009-2010)

Figure 5: Policy, Advocacy and Coordination Operational Expenditures by Type (2005-2006 to 2009-2010)

Figure 6: Policy, Advocacy and Coordination Full-Time Equivalents by Group and Region (2010-2011)

 

List of Tables

Table 1: Program Activity Architecture Accountabilities

Table 2: Expenditures ($M) by Program Activity (2005-2006 to 2009-2010)

Table 3: Preliminary Evidence of Achievement of Policy, Advocacy and Coordination Expected Outcomes (2005-2006 to 2009-2010)

Table 4: Distribution of Preliminary Interviews

 

Acknowledgements

This evaluation was undertaken to provide Atlantic Canada Opportunities Agency (ACOA) management with systematic, neutral evidence on the relevance and performance of its Policy, Advocacy and Coordination (PAC) program activity and to demonstrate accountability for this important function. It is expected that the results of this study will be used to guide the future direction of the PAC program activity with respect to ACOA’s operations. The study was managed and completed by ACOA’s Evaluation Unit.

We would like to take this opportunity to thank the members of the evaluation steering committee for their advice and support throughout this process. Their assistance helped to ensure the relevance and usefulness of this evaluation. Of note is the gracious contribution of two external members: Mr. Rob Greenwood (Director, Harris Centre of Regional Policy and Development, Memorial University of Newfoundland, St. John’s) and Mr. Jim McNiven (Senior Policy Research Advisor, Canmac Economics, Sackville, Nova Scotia).

We are also grateful to the many ACOA staff members, too numerous to acknowledge individually, including staff from the Evaluation Unit, and external key informants who provided their time and essential knowledge in support of this study.

Overall, these contributions were instrumental in building our understanding and correctly conveying the nature of the PAC functions at ACOA.

Marc Lemieux

Director General, Audit and Evaluation (Head of Evaluation)

Atlantic Canada Opportunities Agency

Executive Summary

The Policy, Advocacy and Coordination (PAC) function of the Atlantic Canada Opportunities Agency (ACOA) is central to identifying and effectively responding to opportunities and challenges facing the region’s economy.

The PAC function provides intelligence, analysis and advice to inform and support decision making. The function supports advocacy efforts to influence national policies and programs affecting Atlantic Canada and assists in the coordination of policies, programs and partners to develop and foster collaborative approaches to economic development.

The PAC function is highly decentralized, with units located at head office (HO), in Ottawa, and in each of the four regional offices. ACOA’s Policy Network – which operates at the director general level – is responsible for coordinating and communicating PAC activities occurring across the Agency. The PAC function expends an average of $12.4 million per year, representing approximately 3.8% of ACOA’s total expenditures.

The purpose of this evaluation is to assess PAC’s relevance and performance in achieving its expected result while contributing to the Agency’s strategic outcome. The evaluation covered the five-year period from fiscal 2005-2006 to 2009-2010. In keeping with the terms of reference for the study – which were approved by ACOA’s Executive Committee in April 2011 – the evaluation focused on design and delivery issues and the achievement of short- to medium-term outcomes.

The evaluation methodology included a document and literature review, an analysis of project data, an analysis of seven comparison organizations, 44 key informant interviews (28 internal; 16 external), five focus groups (31 participants), and six case studies (36 key informant interviews). Well over 100 individuals shared their knowledge and insight for this study. Evaluation findings are based on a high level of convergence of multiple lines of evidence and are deemed reliable and valid within the context of the study limitations.

Findings

Relevance

The PAC function is built into the Atlantic Canada Opportunities AgencyAct, which mandates the Agency to make use of program, policy, advocacy and coordination functions to support and promote opportunities for economic development in Atlantic Canada. ACOA’s PAC function is aligned with government-wide priorities and strategies, and is considered unique and complementary to other organizations in the Atlantic region and within the Government of Canada.

The PAC function plays a legitimate and necessary role for which there is an ongoing need and it serves a variety of stakeholders, both internal and external to ACOA. Evaluation findings show that PAC is effective in meeting the needs of external stakeholders, of the minister and of senior management.

Performance

Evaluation results indicate that the decentralized structure of the PAC function, and the activities undertaken by PAC units, are appropriate in achieving intended results. Decentralization allows ACOA to undertake regional economic development “from the ground up” while developing strong working relationships with regional stakeholders in order to better meet their needs and priorities.

Overall, roles and responsibilities across the PAC function are generally established and are consistent with comparative PAC functions. The majority of activities undertaken by PAC units are aligned with the logic model for the PAC function. 

The evaluation shows that ACOA’s PAC function plays an important role in program renewal and the establishment of new programs within the Agency. The PAC function also plays an important role in strategic planning activities undertaken at ACOA since 2006.

The PAC function supports senior management decision making through the conduct of policy research and analysis that is generally deemed to be current, relevant and useful. Such research and analysis is also considered of importance to ACOA Programs and to the Agency’s advocacy and coordination efforts.

Overall, ACOA appears to have adequate capacity to deliver on PAC results.

Areas for Improvement

The evaluation identified areas that can be improved to increase the performance of ACOA’s PAC function, including the following: 

Recommendations

The evaluation team has issued three recommendations, which are further detailed in this report.

The first recommendation seeks to clarify and communicate roles and responsibilities of PAC units while improving coordination, collaboration and communication.

The second recommendation calls for a review of ACOA’s corporate strategic planning process in order to optimally engage PAC resources and knowledge across the Agency.

The third recommendation calls for a review of the PAC function’s performance measurement in order to ensure the accuracy of the PAC logic model and improvements to the Agency’s ability to manage, plan for, measure and report on PAC results.

1. Introduction

The Atlantic Canada Opportunities Agency (ACOA) has a mandate “to increase opportunity for economic development in Atlantic Canada.”[See Footnote i] Within the Agency’s program activity architecture (PAA – Appendix A), the Policy, Advocacy and Coordination (PAC) program activity is one of four that contribute to achieving the Agency’s strategic outcome of a competitive Atlantic Canadian economy.

ACOA’s PAC function is central to identifying and effectively responding to opportunities and challenges facing the Atlantic Canadian economy.

The function provides intelligence, analysis and advice on a range of issues and topics to inform and support decision making by the Agency and its minister(s). PAC supports advocacy efforts to influence national policies and programs that affect Atlantic Canada’s development and interests. PAC also assists in the coordination of policies, programs and partners within the region to develop and foster collaborative approaches to economic development. As such, the PAC program activity permeates all of ACOA’s activities.

In keeping with the Treasury Board of Canada Secretariat (TBS) Policy on Evaluation,[See Footnote ii] ACOA is mandated to evaluate the relevance and performance of all direct program spending every five years. ACOA’s Executive Committee (ExCom) approved the terms of reference for the PAC evaluation in April 2011 on the recommendation of the evaluation steering committee (ESC).

This report presents the results of the evaluation of the PAC program activity. The evaluation examines the relevance and performance of the PAC program activity over a five-year period, from fiscal 2005-2006 to 2009-2010.[See Footnote iii] The evaluation scope excludes the Atlantic Policy Research Initiative (APRI), which was evaluated separately in 2009-2010.[See Footnote iv]

Following the evaluation overview, Section 2 of the report provides a profile of the PAC program activity. Sections 3 through 5 present the evaluation’s findings, organized by broad evaluation questions. The key findings are highlighted throughout these sections. Section 6 presents the general conclusions and recommendations resulting from the study.

Management has agreed with the evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the evaluation’s recommendations, can be found in Appendix F.

1.1. Evaluation Overview

The evaluation was carried out in accordance with ACOA’s approved evaluation plan for 2009-2014, as well as the TBS Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada. The research for this evaluation was completed between July 2010 and November 2011, inclusive of an extensive planning phase, which ran from July 2010 to May 2011.

The evaluation was undertaken by ACOA’s Evaluation Unit. PGF Consultants Inc. was hired to undertake data collection for the focus group component of the study. The evaluation was supported by a steering committee that included representatives from ACOA management and staff as well as external stakeholders. The ESC provided strategic direction, expert advice and guidance regarding the conduct of the study, and it ensured that evaluation issues of importance were addressed. The contributions of the ESC helped to ensure the relevance and usefulness of this evaluation product. 

1.1.1. Evaluation

Rationale

The purpose of this evaluation was to assess the relevance and performance of the PAC program activity in achieving policies and programs that strengthen the Atlantic economy while contributing to ACOA’s strategic outcome. The results of this study fulfill accountability, learning and decision-making needs. It is expected that the findings will be used to guide the future direction of the PAC program activity. The evaluation was done at a program activity level; however efforts were made to highlight regional results where possible.

This is the first formal evaluation of the PAC program activity. An evaluation of APRI was conducted in 2009-2010. The results of that study, summarized in Appendix B, served as contextual information and secondary evidence. 

1.1.2. Evaluation Design and Methodology

The evaluation was designed to address the core evaluation issues of the TBS Policy on Evaluation, which fall into two broad categories: relevance and performance. To address value for money, all evaluations are required to generate clear and valid conclusions on relevance and performance. The following specific questions were developed for each core evaluation issue, based on information gathered and consultations undertaken with senior PAC management and other stakeholders during the planning phase.

Relevance: The extent to which the PAC program activity addresses a demonstrable need and is relevant to ACOA’s mandate, strategic objectives and government-wide priorities and strategies.

  1. Is there a legitimate and necessary role for ACOA’s PAC programming?
  2. To what extent are PAC activities aligned with ACOA’s mandate and strategic outcome, and government-wide priorities and strategies?
  3. To what extent is the PAC program activity meeting the needs of key stakeholders, both internal and external?

PerformanceEffectiveness: The extent to which the PAC program activity objectives have been achieved within the context of expected results and outcomes.

  1. Is the current governance structure for the PAC program activity appropriate? Are there any barriers and challenges to the current structure which may be impacting on the achievement of expected results?
  2. In what manner and to what extent has PAC contributed to setting priorities and strategic direction for the Agency?
  3. In what manner and to what extent has PAC played a role in the development and implementation of ACOA programming?
  4. To what extent is the policy research and analysis undertaken within PAC considered accessible and relevant to decision makers?
  5. Is the coordination of activities related to PAC adequate and effective between: ACOA regional offices, ACOA Head Office (HO) Policy, ACOA HO Ottawa, ACOA ExCom and/or minister, and other levels of government?
  6. To what extent are the activities being undertaken in HO and the regions consistent with the logic model and performance measurement strategy (PMS) for PAC? Are the outputs produced by PAC appropriate to support the achievement of expected results?

PerformanceEfficiency and Economy: The extent to which PAC activities are undertaken in an affordable manner, taking into consideration the relationship between outputs and the resources to produce them. The extent to which resources allocated to the PAC program activity are well-utilized, taking into consideration alternative delivery mechanisms.

  1. To what extent does annual spending on PAC activities reflect planned spending? What processes contributed to the cost variances from planned spending?
  2. How does the PAC program activity compare to other regional development agencies (RDAs) or similar organizations in terms of resources, structure, and other delivery aspects?
  3. Does senior management have access to ongoing information on the performance of the PAC function? If not, what improvements are required to enhance the function’s performance measurement information?
  4. What opportunities exist to improve the efficiency and effectiveness of PAC-related activities and outputs?

In order to answer these questions, a descriptive, mixed-methods research design was chosen involving multiple lines of evidence gathered through the following methods:

The following cases were selected based on input received during preliminary interviews and feedback received from the ESC:

  1. Atlantic Gateway
  2. Atlantic Energy Office
  3. Aerospace and defence advocacy file
  4. Aquaculture advocacy file
  5. Opportunity-based teams approach
  6. Atlantic Population Table

Further details on each case are provided in Section 2.2. The evaluation question matrix and additional details related to each of the above methods can be found in Appendix C and Appendix D. 

1.1.3. Evaluation Limitations and Challenges

The evaluation design and implementation are considered appropriate based on the intended objectives of the study. The use of multiple lines of evidence gathered through a mix of qualitative and quantitative methods helped to compensate for the inherent limitations of each data source and the overall study limitations and challenges, which included:

In light of these challenges, and in keeping with the terms of reference for the study, the evaluation placed a stronger emphasis on design and delivery issues, and the achievement of short- to medium-term outcomes, rather than on the measurement of intermediate and long-term outcomes. While the evaluation relied heavily on qualitative data, multiple lines of evidence were pursued that allowed for triangulation (i.e. convergence of results across lines of evidence) and complementarity of findings (i.e. developing better understanding by exploring different facets of a complex issue).

Well over 100 individuals provided their knowledge and insight to this study, and the findings presented in the report build on a high level of convergence of informed views and opinions. Quotes have been used throughout the report and were chosen to represent the majority of views expressed during key informant interviews, case studies and focus groups.

Within the above context, the results of the PAC evaluation are deemed reliable and valid.

2. Profile of the Policy, Advocacy and Coordination Program Activity

2.1. Context

ACOA’s policy, advocacy and coordination activities have been a central part of the Agency’s operations since its inception in 1987. ACOA’s PAC function stems from the Atlantic Canada Opportunities Agency Act(ACOA Act), which mandates the Agency to “support and promote opportunities for economic development of Atlantic Canada, with particular emphasis on small and medium-sized enterprises (SMEs), through policy, program and project development and implementation, and through advocacy of the interests of Atlantic Canada in national economic policy, program and project development and implementation.”[See Footnote ix] The act also requires that the Agency assist the minister in the coordination of federal government policies and programs with opportunities for economic development in Atlantic Canada. The following points drawn from the document review provide some historical context:

Over the last decade, ACOA has had to adapt to changing political, social and economic forces pressing on the region.[See Footnote x] In the latter part of the decade, PAC’s role has been to support initiatives such as (but not limited to):

2.2. Program Theory

The PAC program activity aims to achieve policies and programs that strengthen the Atlantic economy while contributing to ACOA’s strategic outcome of “a competitive Atlantic Canadian economy.”

While identified as separate program sub-activities in the Agency’s PAA, policy, advocacy and coordination are interrelated and each is influenced by the other. Policy is often at the centre of this relationship because it provides the research, analysis and advice needed to back up advocacy and guide coordination. Advocacy often relies on ACOA’s coordination function to obtain the positions of the Atlantic Provinces and interest groups and to help mobilize regional networks in support of securing opportunities from federal initiatives. The advocacy function, in turn, influences policy and coordination in developing and implementing ACOA programs and priorities to ensure alignment and consistency with federal policy priorities.

Internal stakeholders of PAC include: the ACOA minister and Cabinet, ACOA senior management, program management[See Footnote xii] and officers. External stakeholders include other federal departments and agencies, central agencies, Enterprise Cape Breton Corporation (ECBC),provincial governments, the private sector, communities (e.g. geographic, of interest, sectoral), academic institutions, research communities and non-governmental organizations.[See Footnote xiii]

Figure 1 provides an overview of how the activities, outputs and outcomes of the PAC program activity are expected to contribute to the achievement of ACOA’s strategic outcome. Each of the PAC program sub-activities is described in further detail below.

 

Figure 1: Policy, Advocacy and Coordination Logic Model

(D)

Source: Adapted from ACOA’s Performance Measurement Strategy, June 2010.

Policy

ACOA’s policy function provides a base of understanding for the development of the Agency’s strategic priorities and initiatives, program design, and input to national policy development and federal-provincial relations. This includes policy analysis and advice, the development of policies and frameworks, economic analysis, corporate planning and reporting, as well as research and stakeholder engagement.[See Footnote xiv] It should be noted that policy occurs at both a corporate and a program level. When required, this distinction is highlighted throughout the report.

Through research, analysis and stakeholder engagement, ACOA’s policy function looks at the main opportunities and challenges facing Atlantic Canada’s economy in order to develop the potential of the region’s enterprises and communities. By supporting decision making at the strategic level, the provision of credible, evidence-based policy advice and information leads to Agency policies and programs that strengthen the regional economy. At ACOA, this means supplying information on key national and regional economic issues, primarily to ACOA’s minister, president and senior executives. Strategic policy information can also be provided to other regional ministers, central agencies, the Atlantic caucus and other members of Parliament.[See Footnote xv]

Advocacy

ACOA’s advocacy activities, such as sectoral and horizontal issues, capacity building and agenda management, aim to advance the region’s interests in national policy and program development. Advocacy is used to pursue industrial and regional benefits (IRB) from major Crown projects to improve the position of Atlantic Canadian industries.[See Footnote xvi]

Effective advocacy of Atlantic Canada’s interests is accomplished through environmental scanning, active monitoring and influencing federal government priorities, policy developments and procurement agendas to benefit Atlantic Canada. Advocacy alerts ACOA’s minister, Agency management and other government officials to emerging issues and articulates the interests of Atlantic Canada to government officials (including those at central agencies) and to members of Cabinet in an effort to inform their decisions. In particular, ACOA’s Ottawa office supports the minister’s participation in Cabinet by bringing the Atlantic Canadian perspective to Cabinet decision making. Through regional offices, advocacy mobilizes regional networks such as those involving SMEs within Atlantic Canada’s aerospace industry to respond to federal opportunities. In addition, ACOA’s regional offices are the eyes and ears of the Government of Canada within the regions. They are responsible for developing strategies and sectors, assisting the Minister’s Office as well as participating in Agency advocacy files with HO and Ottawa.[See Footnote xvii]

Coordination

ACOA’s coordination function engages a range of economic partners to address the economic priorities of Atlantic Canada through a coherent and collaborative approach to development, including federal-provincial initiatives, round tables, and expert panels. The Agency also coordinates with other federal departments with regard to regional development efforts, including the Regional Federal Councils.[See Footnote xviii]

Coordination involves building and maintaining relations with other federal departments whose programs affect the development of the Atlantic economy, as well as building and maintaining relationships with other governments (e.g. provincial and municipal) to align priorities and policies with those of ACOA and the Government of Canada. Coordination also involves engagement and liaison with a wide range of private- and public-sector groups and non-governmental organizations with an interest in the development of Atlantic Canada. ACOA’s coordination function aims to ensure that all groups and parties with a responsibility for, or an interest in, economic development in Atlantic Canada work together to minimize duplication of effort and maximize collaboration, contributing to Agency policies and programs that strengthen the Atlantic economy.[See Footnote xix]

While policy, advocacy and coordination are separate program sub-activities, they are interrelated and influence one another. The PAC projects that were chosen as case studies clearly demonstrate the complexity of PAC work, and show the integration between all three elements of PAC. Each case study is described in further detail below:

  1. Atlantic Gateway: PAC played an important role in the development of the Atlantic Gateway – Canada’s eastern portal for international trade – which promotes an efficient connection from the Atlantic region to North America and the rest of the world. PAC used its policy, advocacy and coordination activities to support the development of collaborative relationships among governments and the private sector in this initiative.
  2. Atlantic Canada Energy Office: ACOA has been engaged in supporting the energy sector since the Agency’s inception. In June 2008, the creation of the Atlantic Canada Energy (ACE) Office in St John’s was announced by the then Natural Resources Canada (NRCan) minister. The office was intended as a mechanism for the Government of Canada to coordinate and manage its energy-related activities in the Atlantic Region. The PAC function conducts policy research, advocates on behalf of this office and brings energy players around the table to discuss issues in the Atlantic region.
  3. Aerospace and defence advocacy file: For over 20 years, ACOA has been involved in developing and promoting Atlantic Canada’s aerospace and defence (A&D) industry through a number of tools and strategies. Led by the Ottawa office, PAC’s work on the A&D sector is primarily focused on the application of the IRB policy, which aims to ensure that all regions of Canada benefit from subcontracting opportunities related to major Crown projects.
  4. Aquaculture advocacy file: Despite the aquaculture industry’s history of over 50 years in Atlantic Canada, related policy frameworks had been largely uncoordinated among the provinces and with the federal government. More recent threats from invasive species and disease highlighted other considerable gaps in access to emergency capital. By working closely with stakeholders and playing a pivotal role in bringing the Atlantic Canadian aquaculture perspective to the federal stage, PAC has been able to facilitate positive program and policy changes for the industry as a whole.
  5. Opportunity-based team approach: The underlying intent of the opportunity-based teams (OBT) concept was to use the ACOA Newfoundland and Labrador regional office’s current programs and resources to proactively support areas of importance in the Newfoundland and Labrador economy, and honing in on those where ACOA could help make things happen. The OBT approach relies on an integrated team, capitalizes on the expertise of staff and helps to build community capacity. The PAC group in the Newfoundland and Labrador regional office provides the overall coordination for the OBT approach and the policy research to answer specific needs.
  6. Atlantic Population Table: In 2005, to address the region’s demographic and labour force challenges, the senior officials committee of the provincial and federal governments in Atlantic Canada formed the Atlantic Population Table (APT). ACOA’s role in the APT has been led by PAC and involves providing necessary supporting research, identifying and advocating for regional priorities federally, and coordinating the efforts of APT partners.

2.3. Program Accountability and Governance

Governance and management of the PAC program activity is shared among HO and each regional office.

The Policy unit in Moncton and the Advocacy and Industrial Benefits Unit in Ottawa are both part of ACOA's Policy and Programs Division. The director general (DG), Policy and the DG, Advocacy and Industrial Benefits, both report directly to ACOA’s senior vice-president, Policy and Programs, who is functionally accountable for the PAC function. The PAC structure is decentralized in that each regional office is equipped with policy capacity and each regional DG, Policy reports to the regional vice-president (VP). Accountabilities for PAC, including each of the program sub-activities, are identified in Table 1.

Table 1: Program Activity Architecture Accountabilities

PAA: Program activity – Policy, Advocacy and Coordination

Positions Responsible:

HO – DG Policy

DG, Advocacy and Industrial Benefits

Regional directors general, Policy

PAA: Program sub-activity – Policy

Positions Responsible:

HO – DG Policy

DG Advocacy and Industrial Benefits

Regional directors general, Policy

PAA: Program sub-activity – Advocacy

Positions Responsible:

DG Advocacy and Industrial Benefits

Regional directors general, Policy

PAA: Program sub-activity - Coordination

Positions Responsible:

HO – DG Policy

Regional directors general, Policy

Source: PAA Accountabilities Matrix, ACOA Governance Structure (January 2010).

Atlantic Canada Energy Office

Given the sector’s importance in the development of the region’s economy, PAC activities involving energy have a distinct structure within ACOA. Reporting directly to ACOA’s president, the senior advisor (energy) currently oversees the ACE Office, a joint initiative of the Agency and NRCan. Established in 2009, this office is a mechanism for the Government of Canada to coordinate and manage its energy-related activities in the Atlantic region. Through collaboration and engagement, ACE provides a focal point for federal activities in support of energy sector industry development. As with other ACOA branches, the ACE Office is accountable to the Agency’s president, and reports on joint departmental activities at ACE to both the ACOA president and the deputy minister of NRCan.

Policy Network

The Policy Network is the main mechanism for sharing and coordinating information on policy activities across the Agency. The Policy Network is chaired by the DG, Policy (HO), and includes regional DGs responsible for policy and the DG, Advocacy and Industrial Regional Benefits. Other directors and officers attend meetings by invitation of the chair, as required and as appropriate. The mandate of the Policy Network is to ensure that the Agency’s ExCom receives strategic and timely advice on important policy issues facing the region’s economy and the Agency. The Policy Network harnesses the Agency’s policy capacity in a coordinated team approach in order to share information, to initiate and guide policy activities, and to engage Agency resources in all regions and HO on corporate policy priorities.[See Footnote xx]

The Policy Network works closely with other areas of the Agency as required to ensure issues brought to the ExCom for discussion and decision take into consideration pan-Atlantic and provincial policy implications.

2.4. Expenditure Profile

Table 2: Expenditures ($M) by Program Activity (2005-2006 to 2009-2010)

Program Activity 2005-2006 2006-2007 2007-2008  2008-2009  2009-2010
Enterprise Development 262.3 227.5 213.6 215.0 192.2
Community Development 144.0 146.4 146.1 113.1 155.7
Policy, Advocacy and Coordination   12.4 11.4 11.9 13.3 13.0
Internal Services* 0 0 0 0 42.9
Total Expenditures 418.7 385.3 371.6 341.4 403.8

* Prior to 2009-2010, the cost of internal services was allocated to each program activity using a formula developed and implemented by ACOA finance. Source: Departmental Performance Reports.

From 2005 to 2010, PAC expenditures amounted to $62 million, averaging $12.4 million a year (inclusive of internal services up until 2009-2010, as well as Treasury Board-funded costs and other statutory expenditures) as detailed in Table 2. This represents an average of approximately 3.8% of ACOA’s total expenditures. When the cost of internal services, Treasury Board-funded costs and other statutory expenditures are excluded,[See Footnote xxi] PAC’s expenditures over this period amount to $51.0 million, or $10.2 million per year on average.

Figure 2: Policy, Advocacy and Coordination Expenditures by Region (2005-2006 to 2009-2010)

(D)

Source: GX Financial Data, August 2011.

As can be seen in Figure 2, salaries represent the majority of expenditures (approximately 70%). Activities funded through grants and contributions (BDP and APRI) account for approximately 12% of expenditures, while general operating expenditures account for the balance (approximately 18%). Each is discussed below.

2.4.1. Grants and Contributions Expenditures

The strategic intelligence gained from research informs policy analysis, development and advice on options and strategies. It also provides insight for the Agency’s advocacy efforts and is a source of information for the Agency’s coordination mandate. In addition to policy and economic analyses undertaken internally, the PAC program activity makes use of two grants and contributions programs to support policy research.

APRI, which is a Pan-Atlantic initiative coordinated by HO, was created in 2000-2001 and is one of the main programming tools through which ACOA carries out its policy research responsibilities. Key activities include research studies and reports, support of policy or research partnerships, various types of engagements, including conferences, round tables and consultations, and initiatives that promote public- and private-sector networking on policy-related issues. It also provides a flexible instrument to extend ACOA’s horizontal policy reach by involving a network of regional partners, including independent policy organizations, government departments, universities, colleges and the economics community, business associations, independent researchers and interested groups.

ACOA introduced the BDP in 1995 to provide financial assistance to Atlantic Canada’s SMEs. Utilized by HO and the regional offices, the BDP is another mechanism by which the Agency supports policy research. Its contribution to policy research is engaged, in particular, with developing a better understanding of the Agency’s support for business start-ups and the successful expansion and modernization of SMEs in Atlantic Canada. PAC research activities conducted through the BDP’s business support element include targeted studies on key priority areas such as innovation and commercialization, productivity and competitiveness, trade and investment, skills development, and specific sectors.

From 2005 to 2010, 88 PAC projects were undertaken using APRI (58 projects) and BDP (30 projects) funds, with ACOA providing $7.9 million toward the total cost ($19.6 million) of these projects. Of the 88 projects, 53 were coded as planning and studies (18 BDP, 35 APRI), with ACOA providing $4.6 million toward the total cost ($11.8 million) of these projects. All 35 APRI projects were undertaken through HO. As can be seen in Figure 3, over 60% of BDP planning and studies projects were undertaken through ACOA’s regional office in Nova Scotia.

Figure 3: Number of BDP Projects (Planning and Studies) by Region (2005-2006 to 2009-2010)

(D)

Source: QAccess Data, September 2010

2.4.2. Operations and Maintenance Expenditures

Beyond the salary expenditures that support the PAC function, O&M funds are also used to support policy research projects. According to data provided by PAC management and staff, 69 projects were funded through O&M between 2005 and 2010. While these are mostly studies, they can also include workshops and other advisory services. The expenditure data related to these projects is incomplete and thus cannot be reported. As can be seen in Figure 4, Newfoundland and Labrador and Nova Scotia account for the majority of O&M projects, at 36% each. No O&M projects appear to have been completed in Prince Edward Island.

Figure 4: Number of O&M Projects by Region (2005-2006 to 2009-2010)

(D)

Source: ACOA PAC, February 2011.

According to ACOA’s financial data, PAC program activity operating expenditures (excluding salaries) totalled $9.1 million (net) between 2005 and 2010. As indicated in Figure 5, the highest proportion of operating expenditures relate to professional services and travel, accounting for approximately 80% of total operating expenditures.

Figure 5 shows important differences in the breakdown of operating expenditures by region. Nova Scotia has the highest expenditures in professional services – more than three times that of HO/Ottawa, which has the highest expenditures in travel. Those travel expenditures are more than double that of any other region except Newfoundland and Labrador.

Figure 5: Policy, Advocacy and Coordination Operational Expenditures by Type (2005-2006 to 2009-2010)

Note: Negative values represent the recovery of expenditures that are primarily related to the Federal Councils from other departments. While these funds flow through PAC, PAC functions within each region hold no responsibility for the Federal Council in their region.

Source: ACOA Financial Data, including Standard Objects of Expenditures, August 2011

(D)

 

2.5. Salary Expenditures

From 2005 to 2010, PAC incurred approximately $36.1 million in salary expenditures, representing close to 70% of total PAC expenditures.

As of August 2011, the PAC program activity accounted for 80 full-time equivalents (FTEs); 72 if administrative and library services staff are excluded. This represents approximately 10% to 11% of ACOA’s FTE base (n = 711). Figure 6 provides an overview of the breakdown of PAC FTEs by region, group and level.

Based on data from the Departmental Performance Report (DPR), the number of ACOA FTEs engaged in PAC activities has varied from 75 in 2005-2006 to 116 in 2006-2007 to 95 in 2007-2008 and 89 in 2009-2010. The FTE data provided in ACOA’s DPR for the PAC program activity takes into account an estimate of time spent on PAC-related activities across program sub-activities.

It should be noted that data on projects supported through PAC salary expenditures are not consistently available across the Agency. The evaluation, therefore, cannot report on the number of PAC projects undertaken by staff.

Figure 6: Policy, Advocacy and Coordination Full-Time Equivalents by Group and Region (2010-2011) [See Footnote xxii]

(D)

Source: ACOA PAC, August 2011.

As can be seen in Figure 6, the number of FTEs available to support PAC activities varies by region, with HO accounting for 29% and PEI accounting for 4% of total PAC FTEs. Over half of FTEs fall within the economics and social science services (EC) group, of which 88% are at the EC 06 and EC 07 level.

3. Findings: Relevance

The relevance of ACOA’s PAC program activity was assessed by examining (1) the legitimate and necessary role of ACOA PAC programming, (2) the alignment of PAC activities with ACOA and government-wide priorities and strategies, and (3) the extent to which the PAC program activity is meeting the needs of internal and external key stakeholders.

3.1. Legitimate and Necessary Role for Policy, Advocacy and Coordination

Policy functions play an important role within governments, providing timely, objective and fact-based information to senior managers and ministers to support sound decision making.[See Footnote xxiii] They enable governments to understand and address current and emerging issues, make choices based on the analysis of options and outcomes, and manage resources and programs to achieve specific economic and social objectives affecting the public.

Key Finding:

PAC plays a legitimate and necessary role within ACOA and within Atlantic Canada.

ACOA is responsible for coordinating federal activities relating to economic development in Atlantic Canada and providing a regional perspective on the design and application of national policies and programs.

The creation of ACOA, as well as of the WD and FedNor,[See Footnote xxiv] was announced in 1987 as part of the Government of Canada’s regional development restructuring process. The objective was to decentralize economic development policy and administration away from Ottawa and to encourage better coordination of policies and programming with provincial governments and other local stakeholders.[See Footnote xxv] While CED-Q’s act came into force in 2005, it had been in existence since 1989 under the name Federal Office of Regional Development – Quebec. The commitment to regional economic development continued through the creation of the Federal Economic Development Agency for Southern Ontario and the Canadian Northern Economic Development Agency in 2009.

3.1.1. Policy, Advocacy and Coordination as Outlined in the ACOA Act and the Program Activity Architecture

As previously discussed, the PAC function is built into the ACOA Act, which mandates the Agency to make use of program, policy, advocacy and coordination functions to support and promote opportunities for the economic development of Atlantic Canada.[See Footnote xxvi] The importance of the PAC function within ACOA is further emphasized by its inclusion as one of four program activities in ACOA’s PAA, with each of policy, advocacy and coordination representing a program sub-activity. All three PAC program sub-activities are interrelated and influence one another. While policy work is often at the centre of this relationship, providing the research, analysis and advice needed to support advocacy and to guide coordination efforts, studies demonstrate that policy work is interactive and highly situational.[See Footnote xxvii]

ACOA’s PAC function is considered unique and complementary to other organizations in the Atlantic region and in the Government of Canada. A literature review and key informant interviews reveal there is no other federal department or agency duplicating the functions undertaken by PAC and that overlap is minimal. The comparative analysis indicates that other federal RDAs, including FedNor, have PAC functions that play a similar role within their respective regions.

Policy

Key informants and focus group participants agree that ACOA has an ongoing need for timely, evidence-based information and analysis to support sound decision making at all levels of the organization, whether related to policy advice, environmental scanning, strategic planning, operational planning, program planning, development or delivery. They state that there is a need for region-specific as well as pan-Atlantic information and analysis. PAC provides research at both levels.

The importance and need for pan-Atlantic research in particular was confirmed by the 2010 evaluation of APRI, a program PAC administers to support Atlantic policy research and engagements.[See Footnote xxviii] This evaluation also states that research undertaken through APRI complements other research conducted by provincial governments, universities and think tanks in Atlantic Canada as well as national organizations such as the Conference Board of Canada, Statistics Canada, and the Policy Research Initiative. The extent to which these needs are met is discussed in section 3.3.

Advocacy

PAC’s Ottawa office is focused on strategic advocacy activities. In particular, the Ottawa office supports the ACOA minister’s role in bringing the Atlantic Canada perspective to Cabinet decision making. In the words of one key informant, advocacy is “daily liaison with other federal departments when they are developing their policies to ensure that they know Atlantic Canada’s interests and that [the region] is reflected in [the] policies.” PAC develops important relationships with other federal departments to ensure regional economic development interests and priorities are represented in federal policies, programs and regulations.

In addition, ACOA’s regional offices serve a regional intelligence-gathering function for the Government of Canada within the Atlantic provinces. They are responsible for developing strategies and sectors, assisting the Minister’s Office, and participating in Agency advocacy files with HO and Ottawa.

Interviews and case studies highlight the key role of advocacy in ACOA’s priorities, including procurement initiatives related to A&D, the Atlantic Gateway and aquaculture. For example, PAC’s A&D work concentrates on the implementation of the federal IRB policy to enhance opportunities for Atlantic Canadian companies to benefit from major Crown projects.

Coordination

Coordination is a natural extension of both policy and advocacy activities. In his most recent report to the Prime Minister on the Public Service of Canada, the Clerk of the Privy Council highlighted the role of collaboration on policy excellence, noting the complexity of issues and the need to work horizontally and with a variety of governmental and non-governmental stakeholders.[See Footnote xxix] With four provincial governments in Atlantic Canada, as well as countless other important public- and private-sector stakeholders, PAC’s coordination activities are critical to the development of integrated and coherent approaches for addressing economic development challenges and opportunities.

Interviews, focus groups, case studies and a literature review indicate that PAC units develop important relationships and play important coordination roles while collecting key intelligence from external stakeholders. In particular, regional offices are responsible for coordinating federal councils, various committees, and meetings between different government departments as well as providing information to the Privy Council Office. Key informants also state that with offices throughout the region and involvement in a wide variety of initiatives aimed at promoting economic development, ACOA is recognized by many stakeholders as the main federal presence in Atlantic Canada.

3.2. Alignment with ACOA and Government-wide Priorities and Strategies

As presented in Figure 1, PAC directly supports ACOA’s mandate through its three interrelated program sub-activities. According to a document review, case studies, focus groups and key informant interviews, PAC leads and supports strategic policy, advocacy and coordination activities that are aligned with both the mandate and priorities of the Agency. For instance, a review of research studies undertaken or supported by PAC shows that topics fit with Agency priorities and were related to innovation and commercialization, trade and investment (currently referred to as international business development), entrepreneurship and skills development, community economic development, economic policy research, and key sectors in the region, including energy, aquaculture and A&D.

Key Finding:

PAC is aligned with ACOA’s mandate and strategic outcome, as well as government-wide priorities and strategies.

In addition, case studies illustrate how PAC supports ACOA’s involvement in key sectors that promote economic prosperity in Atlantic Canada. For example, the Atlantic Gateway has enhanced economic development through trade opportunities and the creation of jobs, including those related to various infrastructure projects at airports and ports and with roads. The growth of the aquaculture industry has led to the creation of skilled jobs with higher earned incomes in rural areas that have few other employment opportunities.

While the case study of the Atlantic Population Table shows that PAC has played an important role in the initiative, the APT was indirectly related to ACOA’s mandate, especially in its early years when it focused on increasing immigration and population retention in the Atlantic region. This strategy was viewed as important for enhancing the workforce across Atlantic Canada, leading to greater earned incomes and employment opportunities. Over time, it evolved to focus more closely on skills, bringing it more directly in line with ACOA’s mandate.

ACOA’s mandate and PAC’s key result are linked to one of the federal government’s 16 outcomes – strong economic growth – which is focused on increasing economic growth and development in all regions and all sectors of the economy. Over the period of this evaluation, the Government of Canada has clearly been focused on economic recovery and growth.[See Footnote xxx] PAC’s outcomes are linked and responsive to the federal government’s priorities and strategies including the Whole-of-Government Framework, Advantage Canada and the CEAP.[See Footnote xxxi]

ACOA has played a key role in implementing the CEAP in the Atlantic region. The PAC function has helped to ensure that the CEAP funding made available through the Community Adjustment Fund (CAF) and the Recreational Infrastructure Canada (RInC) Program was useful and relevant to communities in Atlantic Canada by providing intelligence, analysis and advice while also coordinating policy efforts.[See Footnote xxxii]

The finding that PAC is aligned with ACOA and federal priorities is also supported by the 2010 APRI evaluation. This study reports that APRI research and engagement projects “are focused on the priorities of the Government relating to economic development. They support the PAA, especially in terms of policy decisions and direction.”[See Footnote xxxiii] All APRI files were aligned with ACOA’s areas of interest: international trade, labour market and skills development, productivity and competitiveness, natural resources, rural and urban issues, innovation, trade corridors, transportation, immigration, strategic sectors and aboriginal development.[See Footnote xxxiv]

3.3. Extent to which Stakeholder Needs Are Met 

PAC operates within a complex environment and serves a variety of stakeholders, both internal and external. Key informant interviews and case studies reveal that stakeholder needs vary but are aligned with PAC’s three program sub-activities of policy (i.e. information, research, analysis and advice), advocacy (i.e. representation of regional needs) and coordination (i.e. facilitation of relationships and collaborative action). PAC uses both informal (e.g. meetings, contact with stakeholders, officer level meetings) and formal (e.g. consultations, bilateral meetings, client surveys, OBT) mechanisms to identify the needs of its stakeholders.

External Stakeholders

Key Finding:

PAC is effective in meeting the needs of external stakeholders.

External key informants and case study interviewees state that PAC provides important policy research and analysis as well as coordination support. For example, other federal government departments and the Privy Council Office often depend on PAC to provide regional context and information to meet their specific needs. Case studies on the Atlantic Gateway and aquaculture stress the importance of PAC bringing a pan-Atlantic vision to the table. The decentralized structure of PAC assists in developing strong working relationships and meeting stakeholder needs, particularly at the provincial level. In the words of one key informant: “The regional offices are far more in touch with the needs, issues and opportunities [of stakeholders] than would be a head office function on its own, so I suggest that the decentralized approach has merit.”

Internal Stakeholders

Key Finding:

PAC is effective in meeting the needs of the minister and senior management.

PAC has a central role in supporting the needs of the minister and ACOA’s senior management. Interviewees indicate that PAC is available and responsive to requests from the minister and senior managers. Over the five-year period covered by this evaluation, PAC units across the Agency have tackled issues spanning both ED and CD program activities. Focus group results confirm that PAC has played a significant role in program renewal within the Agency.

As mentioned above, key informants and focus group participants agree that ACOA has an ongoing need for timely, evidence-based information and analysis to support sound decision making at all levels of the organization, whether related to policy advice, environmental scanning, strategic planning, operational planning, program planning, development or delivery. While all these activities are not explicitly captured in the PAC logic model, according to key informants they are addressed in the day-to-day activities of the regional PAC units.

Key Finding:

Programs and internal stakeholders identified needs that are not being met consistently by PAC units.

Key informants and focus group participants identified gaps related to their need for timely policy information, analysis and advice. As a result, some focus group participants indicated that they conduct their own program policy research. The Newfoundland and Labrador regional office stood out in the focus groups, key informant interviews and the OBT case study as having a more effective, integrated relationship between Programs staff and PAC compared to other regions.

4. Findings: Performance – Effectiveness

The evaluation of the effectiveness of ACOA’s PAC program activity was assessed by examining (1) evidence of achievement of PAC’s expected results, (2) the effectiveness of the current PAC governance structure, (3) the contributions of PAC to setting priorities and strategic direction for the Agency, and (4) the accessibility and relevance of policy research and analysis.

4.1. Achievement of Expected Results

The PAC logic model (Figure 1) provides an overview of how the activities and outputs of PAC are expected to contribute to the achievement of expected results.

While the evaluation places a stronger emphasis on design and delivery issues, Table 3 presents preliminary evidence of the achievement of expected results drawn from the case studies. These case studies were critical in highlighting and profiling the interrelationship between policy, advocacy and coordination and their influence over the life cycle of these initiatives.

In addition to the evidence in Table 3, PAC is clearly acknowledged as having played an important role in establishing and implementing major programs such as the Atlantic Innovation Fund and the CAF and RInC programs, and it has contributed to program renewal within the Agency.

Key Finding:

PAC plays an important role in program renewal and the establishment of new programs within the Agency.

Table3: Preliminary Evidence of Achievement of Policy, Advocacy and Coordination Expected Outcomes (2005-2006 to 2009-2010)

Policy program sub-activity expected outcomes: Well-informed policy decisions reflecting opportunities and challenges of the Atlantic region’s economy while considering enterprise and community development potential.

Evidence of Achievement[See Footnote xxxv]

1. Atlantic Gateway - PAC-led policy research and a business case supported agreement on a 2007 federal-provincial memorandum of understanding and the 2011 Atlantic Gateway Strategy.

2. Atlantic Population Table - PAC provided relevant research and evidence to support the development of the APT, which led to a federal-provincial Atlantic population strategy.

3. Aquaculture - ACOA partnered with the Department of Fisheries and Oceans in the development of a government response to the task force on “Fostering a Sustainable Salmon Farming Industry for Atlantic Canada.”

Advocacy program sub-activity expected outcomes: Atlantic ED and CD interests are reflected in emerging and changing federal economic policies, programs and regulations.

Evidence of Achievement

1. Aerospace & Defence - PAC’s procurement advocacy activities have helped Atlantic SMEs become integrated into the global supply chain. It has resulted in an estimated $2 billion in opportunities for Atlantic SMEs.

2. Atlantic Canada Energy Office - ACE advocacy efforts contributed to the approval of four Atlantic projects under the Clean Energy Fund in 2010 nearly 30% of national funding.

3. Aquaculture - PAC advocated for a joint government response to the salmon farming crisis in the region, resulting in an announcement of $20 million in assistance to New Brunswick operators.

4. Atlantic Gateway - PAC’s advocacy function was critical to the Atlantic Gateway’s inclusion in the national gateways agenda and infrastructure commitments of over $200 million.

Coordination program sub-activity expected outcomes: Coordination of partners in addressing the economic priorities of Atlantic Canada through a coherent approach to development.

Evidence of Achievement

1. Atlantic Population Table - PAC played an important role in bringing together federal and provincial partners as part of the APT, leading to collaborative action on shared priorities among members.

2. Atlantic Canada Energy Office - The ACE Office facilitated greater collaboration and co-operation among federal and provincial governments and industry; its work led to agreement on the Atlantic Energy Gateway.

3. Aerospace and Defence - PAC supported the formation of the regional A&D industry association that includes corporate executives and the four Atlantic Provinces as well as ACOA. The association has positioned the A&D industry with one coordinated regional voice.

* Evidence of achievement has been extracted from six case studies completed as part of the PAC evaluation.

4.2. Effectiveness of Governance Structure

The Policy on Management, Resources and Results Structure defines governance as “the processes and structures through which decision-making authority is exercised.”[See Footnote xxxvi] In the public sector, the principles that guide action are strategic vision, values and ethics, transparency in decision making, collaboration, and clear accountability. In practice, successful public-sector governance requires attention to expected results, to clearly defined roles, responsibilities and relationships among decision-makers as well as among public-sector organizations, political executive and principle stakeholders, and to well-defined accountabilities.[See Footnote xxxvii]

Key Finding:

PAC’s decentralized structure is appropriate and considered essential for regional economic development.

As with other RDAs, including FedNor, ACOA’s PAC function has a highly decentralized governance structure, with units located in HO, Ottawa, and each of the four regional offices. Key informant interviews and case studies clearly support PAC’s decentralized structure. Key informants spoke of the importance of regional development coming “from the ground up” and of each regional office being able to focus on the needs and priorities specific to its province and local areas.

The need for region-specific policy research is supported by several Canadian reports that examine policy research in the public sector.[See Footnote xxxviii],[See Footnote xxxix] Canada has distinct regional identities, and many issues require policy research at a provincial or community level.

Both key informant interviews and case studies illustrate that PAC develops close working relationships at provincial and local levels. For example, case studies demonstrate that each PAC regional office has developed relationships with local stakeholders related to aquaculture, A&D and the Atlantic Gateway. Key informants speak of the unique situations of each province and the need for timely intelligence and research that helps address specific needs and opportunities.

Key informants and case study respondents identify challenges to the decentralized structure. The most notable issue is the risk of overlap and duplication of PAC activities between regions, including Ottawa, and between PAC and programs.

Key Finding:

PAC’s decentralized structure increases the risk of overlap and duplication of activities across PAC units.

The lack of descriptive data (e.g. number, type, subject, audience) on the outputs being produced by the PAC units does not allow the evaluation to fully assess the degree of overlap or duplication within ACOA’s PAC structure. Key informants, however, emphasize this as an important risk.

4.3. Contribution to Priority Setting and Strategic Direction

Strategic planning is an important element of good management practice. It is a process that focuses on the future and sets priorities that help an organization make decisions about the best use of its resources.[See Footnote xl] A successful strategic planning process establishes both top-down direction and bottom-up identification of issues, and ensures that the development of options reflects an appreciation of priorities, gaps and risks affecting the department as a whole, while being operationally realistic. [See Footnote xli]

The Fifth Report of the Prime Minister’s Advisory Committee on the Public Service recognizes that one of the Public Service’s most important functions is maintaining a capacity for strategic thinking and policy advice. To this end, the public service should continue to invest in the sustained examination of issues beyond the current agenda and in developing people with the skills to do this kind of work. It should engage other sectors and other jurisdictions on a continuing basis to understand emerging trends in the domestic and global environment and pay particular attention to the emergence of new ways of adding value, as well as to changes in how knowledge is transmitted in the global economy.[See Footnote xlii]

In 2006 ACOA began a strategic planning process intended to set strategic priorities and direction for the next three to five years.[See Footnote xliii] This process was fed by the work of PAC and by the senior executive retreats involving ACOA’s minister, board members and senior management team. Following the 2006 events, the Policy Network initiated a process in June 2007 to identify new policy priorities for ACOA and to examine their possible impact on the Agency. Five priorities (i.e. productivity and competitiveness, labour market and skills, science and technology, natural resources and knowledge economy, and urban-rural) were identified and policy priority papers outlining the importance of the priorities were developed. These papers supported the discussion at the last formal executive retreat in October 2007 and provided context for program renewal.

Key Finding:

PAC is viewed as having a lead role in strategic planning and priority setting for the Agency.

Between 2008 and 2010 the Agency focused its energies on program renewal and the implementation of programs resulting from the CEAP. While PAC’s contributions to these exercises were critical, key informants and focus group participants argue that there is a need for a strategic planning process that is facilitated, inclusive and reflects a high level of engagement, noting that an improved planning process would improve coordination and efficiencies. Key informants, including several senior managers, expressed a desire for HO PAC, in particular, to take a leadership role in this regard.

In addition to the strategic planning activities outlined above, ACOA has continued to foster its integrated business planning process and prepares annual operational plans to support work plans, budgets and resource allocation. PAC regional units play an important role in supporting these processes.

Key Finding:

ACOA’s strategic planning process could be improved.

It should be noted that a strategic plan for ACOA was developed and approved in November 2011, but its development occurred outside the scope of this evaluation.

4.4. Accessibility and Relevance of Policy Research and Analysis

Overall, key informants indicate that policy research is moderately to largely relevant and useful for decision making. Interview results with senior management were positive, indicating that policy research is current, relevant and useful to them.

Key Finding:

In general, policy research is deemed current, relevant and useful to decision making.

A document and literature review indicate that, to be effective, policy research should be strategically focused, based on stakeholder needs and aligned with the strategic direction of the organization.[See Footnote xliv]

Evaluation results indicate that, apart from regional priority setting, there is no formal process for the establishment, coordination and communication of policy research priorities that reflect ACOA’s strategic direction and feed into its strategic planning process.[See Footnote xlv]

Key informants acknowledge that policy research needs to be reactive to the needs of the minister and senior management. At the same time, key informants and focus group participants indicate a need for resources to be directed toward proactive work that addresses longer term, strategic issues. PAC set a priority in 2006 of providing intelligence and information that was both proactive and reactive.[See Footnote xlvi] The 2010 APRI evaluation recommended the implementation of more inclusive processes in setting priorities and selecting projects, as well as a need for APRI management and staff to work more closely with ACOA programs and regional staff to identify research needs.

According to a document review and key informant interviews, PAC-led and/or supported strategic policy research is important to the Agency in supporting programs as well as advocacy and coordination efforts. Results achieved through key sector initiatives such as the Atlantic Gateway and the APT highlight the importance and need for readily available, strategically focused and relevant policy research and analysis.

Key Finding:

Timely, strategically focused and relevant policy research is important to programs, and to ACOA’s advocacy and coordination efforts.

As part of the Atlantic Gateway, PAC provided ongoing policy research and advice to senior managers and other stakeholders related to economic development issues and opportunities. In particular, the Atlantic Gateway business case was critical to the initiative moving forward. Several key informants noted the importance of the Atlantic Gateway business case as a tool to advocate within the federal government and beyond, and to guide the entire project.

ACOA PAC was involved in the foundation of the APT by providing policy research on immigration that supported the premise that population issues were highly relevant to regional economic development. Such early efforts led the Agency to co-operate with other federal and provincial departments, identifying the need for a federal-provincial Atlantic population strategy. Interviewees describe one of PAC’s critical contributions as providing relevant research and evidence in support of the development of the APT.

It is estimated that 53 policy research studies categorized under the planning and studies program sub-element field in QAccess were conducted through APRI (35) and the BDP (18). Furthermore, 69 research projects were supported through O&M funding from 2005-2006 to 2009-2010.[See Footnote xlvii] In addition to G&C and O&M projects, PAC also conducts various research papers, such as sector and economic analyses, for internal purposes, but there is no record of the number and scope of these outputs.

Key Finding:

There is a need for a formal process to establish, coordinate and communicate policy research priorities.

In 2010-2011 an Agency-wide initiative to develop a policy research agenda for the coming two or three years was undertaken. It is, however, too soon to assess the impact of these efforts in addressing the concerns raised in this evaluation.

5. Findings: Performance – Efficiency and Economy

For the purposes of this study, efficiency and economy were assessed by examining (1) the existence of PAC structures and mechanisms supporting efficiency and economy, (2) evidence that these structures and mechanisms have resulted in efficiency and economy gains for PAC or ACOA overall, and (3) evidence that PAC has identified and pursued opportunities for improvement and best practices.

5.1. Mechanisms that Support Efficiency and Economy

Efficiency and economy are concerned with the ongoing, dynamic process of optimizing resources and activities. Such optimization is facilitated by the existence of strong management structures, practices and mechanisms. The ability to operate in an efficient and economical manner, therefore, depends on, but is not limited to, the existence and effectiveness of governance structures, roles and responsibilities, coordination and communication mechanisms, performance measurement and reporting.

5.1.1. Governance, Roles and Responsibilities

As previously mentioned, ACOA’s PAC function has a highly decentralized governance structure with units located in HO, Ottawa and each of the four regional offices. As previously mentioned, key informant interviews and case studies clearly support this decentralized model and its appropriateness to regional economic development.

To maximize effectiveness, efficiency and economy, decentralization requires that roles and responsibilities be clearly established, communicated and implemented within the system. As such, the evaluation assessed the validity of the logic model and examined understanding of roles and responsibilities among PAC units.

Key informant interviews, focus groups, case studies and a document review provide evidence that PAC’s roles and responsibilities are generally established, and the majority of activities are aligned with the PAC logic model (Figure 1). When compared across RDAs, including FedNor, the activities and outputs of the PAC functions are similar.

Key Finding:

The activities and outputs of PAC functions across RDAs and FedNor are similar.

Key informants are aware, in general terms, of the different PAC roles across the PAC units. In particular, key informants indicate that the HO PAC unit is more focused on addressing pan-Atlantic issues, supporting corporate priorities and the president, providing information to support the advocacy efforts of the Ottawa office, Atlantic Gateway and ACE Office. Key informants are also aware that Ottawa PAC supports the ACOA minister(s) and leads advocacy efforts, including the IRB.

Key informants and case study interviewees are aware of the regional offices’ involvement in policy, coordination, planning and, to a lesser extent, advocacy. They highlight the importance of the federal-provincial coordination role played by regional VPs, and the province-specific intelligence and contacts that the regional PAC units provide to the minister and central agencies.

For most of the scope of the evaluation, PAC regional offices were responsible for integrated and operational planning within their respective regions and for contributing to corporate reporting activities (e.g. the RPP and DPR), which were coordinated by Corporate Planning and Performance Measurement. As of fiscal year 2010-2011, HO PAC assumed responsibility for ACOA’s corporate planning and reporting functions, with HO Policy coordinating an integrated business planning process for the Agency. Policy staff within the regional offices are active participants in this process and use the resulting integrated business plan as the foundation for their support to regional operational plans.

Key Finding:

There is a need to clarify and communicate PAC roles and responsibilities in a number of areas.

While the above areas reflect common understanding of PAC roles and responsibilities across the Agency, all lines of evidence point to the need to improve clarity and communication of roles and responsibilities in the following areas:

5.1.2. Coordination and Communication Mechanisms

Stakeholder Coordination and Communication

Key informant interviews and case studies indicate that PAC’s external coordination and communication activities are generally well-defined and implemented. Overall, they reveal that PAC has strong relationships with external stakeholders, particularly at the regional level. 

Key Finding:

PAC’s external coordination and communication efforts are considered effective.

Most relationships with external stakeholders involve informal mechanisms, including telephone conversations, e-mails and meetings. Several key informants recommend regular contact between PAC and external stakeholders to ensure timely sharing of information that could assist in taking advantage of particular opportunities or developments. While some formal mechanisms exist to aid external coordination (e.g. Federal Councils, Atlantic Population Table), these could be strengthened to ensure that PAC is able to maximize its involvement in opportunities for economic development.

Coordination and Communication within PAC

In contrast to the strengths identified relating to external coordination and communication, a number of issues were identified relating to PAC coordination and communication within the Agency. These findings are consistent with WD’s evaluation of its PAC function.[See Footnote xlviii]

Several informants reported that information does not flow effectively to and from regional offices, HO and Ottawa. Key informants stated that coordination between regional offices and with HO could be improved. Enhancing information sharing and coordination would limit duplication and improve efficiencies, and allow for regional offices to be engaged in the process of identifying policy research to ensure that advocacy efforts are supported by required data and analysis. It would also ensure the optimal use of ACOA’s rich base of knowledge and skills. The recently launched policy research agenda was cited as a positive initiative for improving coordination and communication.

Key Finding:

There is a need for improved coordination and communication across PAC units to reduce the risk of overlap and duplication and to optimize the use of ACOA’s specialized knowledge and skills base.

The mandate of ACOA’s Policy Network is to harness ACOA’s PAC capacity in an integrated team approach to share information, initiate and guide policy activities, and engage Agency resources in all regions and at HO on corporate policy priorities.[See Footnote xlix] Its role is to act as a coordinating body for Policy activities at HO, in regional offices and in the Ottawa office.[See Footnote l] This mechanism for coordination and communication is focused at the DG level. The Policy Network is recognized by key informants as having the potential to be an effective mechanism for facilitating communication and coordination; it is not, however, considered effective for a number of reasons. Some key informants are concerned that poor information flow from the Policy Network to operational level PAC staff is impacting on the effectiveness and efficiency of PAC. In addition, key informants raised a number of issues that impact the extent to which Policy Network can achieve its objectives (e.g. irregular meetings, member participation, action follow-up).

Key Finding:

There is a need to improve the effectiveness of ACOA’s Policy Network.

Key informants and focus group participants agree that there are opportunities for better coordination, especially on strategic sectors. Several recommend that leads be designated on particular sectors to ensure better sharing of information, efficient use of specialized knowledge and skills from all parts of the Agency and a more coordinated approach. The aquaculture and Atlantic Gateway case studies, both past champion files, illustrate the benefits of having clearly identified functional leads with specialized knowledge and networks.

Coordination and Communication between PAC and Programs

Focus groups, key informant interviews and case studies showed that communication and coordination between PAC and Programs could be strengthened. Overall, PAC involvement with Programs is not consistent across PAC units.

Key Finding:

There is a need for improved communication and coordination between PAC and Programs.

As previously mentioned, the Newfoundland and Labrador regional office was highlighted as having a more effective, integrated relationship between Programs staff and PAC compared to other regions. In Newfoundland and Labrador, the OBT approach has led to an integrated relationship between Programs and PAC staff and demonstrates an effective method for managing projects. This process relies on an integrated team approach between Programs and PAC, with timelines for achieving results.[See Footnote li] The approach has a set process with clearly defined steps, documents to be developed and players to be involved. According to the lessons learned section of the Report on Change Management Strategy: Opportunity-Based Teams, individuals involved in OBT note that it is important to follow the established process in order for OBT to work effectively.

The OBT case study illustrates how PAC staff members collaborate with Programs through all stages of program delivery. Key informants note that some other regions are starting or considering the use of an OBT or similar integrated team approach. These initiatives were not sufficiently established to be considered in this evaluation.

Coordination and Communication of Policy Research

Mechanisms to share and coordinate policy research and analysis at the operational level are mostly ad hoc and informal. According to key informants and focus groups, there is currently no centralized database of policy research or a formal structure to communicate research and analysis at the Agency. This is consistent with the findings of the APRI evaluation and WD’s PAC evaluation.

Key Finding:

The absence of effective coordination and communication mechanisms for policy research has resulted in a lack of awareness of work undertaken across the Agency and increased the risk for overlap and duplication of effort.

However, it was noted during interviews that formal mechanisms for communication and coordination of policy research and analysis at a senior level exist. Examples of these are the Policy Network, ExCom, and the DG Programs/Operations. These mechanisms support the effectiveness of PAC and its responsiveness to the needs of senior management.

Due to the lack of formal functional mechanisms to communicate and coordinate policy research at the Agency level, there is a risk that efforts are duplicated and that impact is not maximized. While there is insufficient information to show clear overlap or duplication of policy research conducted, key informants and focus group participants expressed concern that it is possible that similar sectors are being explored in different regions with limited or no communication and/or coordination. As stated before, there is a lack of descriptive information on internal research papers outside of G&C and O&M projects, thus making it difficult to determine the degree to which overlap or duplication is an issue.

In response to the APRI evaluation a Policy Research Working Group and SharePoint[See Footnote lii] site communicating policy research were recently launched. These are signs that processes are becoming more formalized. The implementation of these tools is outside the scope of the current evaluation, and it is too early to see results.

5.1.3. Performance Measurement and Reporting

Measuring, monitoring and reporting on performance are at the foundation of results-based management. This life-cycle approach to management integrates planning, measurement, reporting, learning and change, with the goal of improved results for Canadians.

At ACOA, having PAC as a program activity within the Agency’s PAA brings with it specific central Agency requirements for performance measurement and reporting.

Key Finding:

Most comparison departments/agencies do not include policy functions in their PAAs. While PAC is included in RDA PAAs, CED-Q is planning to remove it as of 2012-2013.

CED-Q and WD’s PAC function are also included in their respective PAAs. FedNor does not have its own PAA and instead falls under that of Industry Canada. The other government departments examined in the comparative analysis do not include their policy function as a separate program activity within the PAA.[See Footnote liii]

Given the nature of PAC activities, outputs and outcomes, the measurement and reporting of PAC results is a significant challenge. All comparative organizations describe difficulties with performance measurement of policy functions in general, with particular challenges related to the development of appropriate and relevant quantitative performance measures for PAC. Most key informants expressed the concern that quantitative data alone cannot accurately convey the PAC performance story.

Key Finding:

Performance measurement for policy functions such as PAC is a common challenge among other federal departments and agencies.

The comparative analysis shows that for most organizations with less formal reporting requirements for their policy activities, performance measures are less standardized, are implemented at an operational level rather than at the branch level (i.e. employee performance measures such as number of research reports written per year or number of consultations held), or are simply non-existent. Although there are no formal PAC activity reporting requirements for the international comparative Highlands and Islands Enterprise Development, official performance indicators and targets nonetheless exist (e.g. completion of research projects, uptake of research produced).

Attribution of results to specific PAC activities also represents an important performance measurement challenge, particularly for advocacy and coordination-type activities, which are often long-term, create less tangible outcomes, and typically involve a large number of partners. Such challenges are considered common to the development and use of performance measurement in regional development policy.[See Footnote liv]

Although no interviewees felt that the measurement and reporting of PAC results is altogether infeasible, FedNor reports that Industry Canada repositioned many key policy areas under Internal Services as of 2009 in part to simplify and streamline reporting. CED-Q is also planning to remove PAC from its 2012-2013 PAA in order to acknowledge its horizontal nature and alleviate measurement and reporting challenges.

Key Finding:

PAC performance measurement data is insufficient to support results-based management of the PAC function.

While a number of output and outcome indicators for PAC are identified in the ACOA Performance Measurement Framework, key informants consulted during the planning stage of this study stated that data related to these indicators are either unavailable or are inconsistent across regions. In addition, while a PMS exists for PAC, it has yet to be fully implemented. Key informants also reported a lack of awareness among PAC staff of the existence of this strategy and of its requirements. This results in limited performance measurement data related to outputs and expected outcomes of PAC, which limits the degree to which PAC results can be managed, planned for, measured and reported. Key informants highlighted the need to develop and make available tools and systems for tracking and monitoring project data, including G&C and O&M funded projects. This issue was also reported in the APRI evaluation and WD’s evaluation of its PAC function. While recent examples of output tracking[See Footnote lv] implemented by Nova Scotia, Newfoundland and Labrador and New Brunswick appear promising, these efforts are not being implemented in a coordinated fashion across the regions.

It should be noted that a methodology and template for generating qualitative reviews of PAC projects and/or initiatives based on high-level outcomes was developed and resulted in 23 reviews between 2008-2009 and 2009-2010. While these reviews were useful as supporting evidence for the evaluation, particularly for case studies, template elements related to outcomes and impacts were incomplete and key informants were unable to speak to their use or usefulness as an ongoing performance measurement tool for PAC.

5.2. Evidence of Efficiency and Economy

Key Finding:

The capacity to deliver on PAC results is considered adequate by ACOA staff.

Though variations in resources and position classifications exist, most internal interviewees noted sufficient capacity to deliver PAC activities, and many argued that capacity would be sufficient given better coordination and communication. At the same time, some key informants argued that PAC is not sufficiently resourced to be proactive and innovative; and others noted the increasing burden of corporate planning and reporting. Several key informants argued that PAC staff are “highly skilled and experienced people.” but in order to maximize the use of their knowledge and skills, the Agency needs to better coordinate their work under a shared vision and direction. Similar comments were also made during focus groups with Programs staff.

5.2.1. ACOA Policy Advocacy and Coordination Versus Comparable Organizations

Based on DPR data for 2009-2010, ACOA’s PAC budget was $13 million, which accounted for 3.8% of ACOA’s overall budget. WD’s PAC budget was $8.9 million and CED-Q’s, $5.6 million, accounting for 3.8% and 1.9% of overall budget, respectively.[See Footnote lvi] For this same time period, PAC FTEs accounted for 12.4% of ACOA’s overall FTEs, compared to 14.3% for WD and 11.5% for CED-Q.

Differences in geographic mandate and program delivery models may account for some of this variance. For example, ACOA and WD both have four provincial/regional offices, while CED-Q had 14 regional offices in Quebec at the time of evaluation. ACOA and CED-Q deliver program funding to a range of both commercial and non-commercial clients, while WD delivers almost all programming to SMEs indirectly via partnership agreements with the provinces and other partners. In addition, differences in existing provincial policy capacity may help explain some of the differences in resources.

While there are no established benchmarks for the proportion of policy-related resources per provincial region covered, or evidence that policy activity related to commercial programming is more resource-intensive, these dimensions of structure and programming may play some role in explaining the PAC resourcing variance.

In addition, it is important to note that the comparative analysis did not include an examination of outcomes or the quality and quantity of activities undertaken by each comparative.

5.3. Best Practices

Several region-specific best practices were raised that could support increased efficiencies, economy and effectiveness of the PAC function. These include:

In addition, the PAC unit in ACOA’s regional office in New Brunswick has taken a proactive role in the management of its policy-based information and research by using SharePoint to store its documents. Early adoption and implementation of SharePoint has allowed the unit to improve productivity by more easily sharing policy-based information among colleagues. “In particular, the management of our ministerial correspondence by our New Brunswick corporate secretariat has been recognized as an innovative solution.” Other than the policy research SharePoint site, there is no indication of groups organizing information in the same way as the New Brunswick PAC.

The Newfoundland and Labrador regional office’s OBT approach was also identified by several key informants as a best practice. The underlying intent of the OBT concept was to use the current programs and resources to proactively support areas of importance in the provincial economy and hone in on those where ACOA could help make things happen. The PAC group in Newfoundland and Labrador provides the overall coordination for the OBT approach and the policy research to answer specific needs.

Within the OBT approach, several best practices[See Footnote lvii] were highlighted, and it is believed the following are key elements to the approach’s success:

Presentations on the OBT approach have been given to other regions, notably Nova Scotia and Prince Edward Island; however these regions had not established a similar process at the time of this evaluation.

The Agency’s New Brunswick office uses a regional priority approach instead of an opportunity-based approach. A research paper is developed for each priority, titled “In Focus.” Each research paper looks at one New Brunswick priority and describes ACOA’s possible involvement. It also looks at the different players within the Agency, more particularly “it’s looking where the connections are in the Agency between Policy and Programs.” The In Focus papers can be found on the New Brunswick regional office’s intranet site. Other regional offices investigate their own sectors, but New Brunswick is the only region that has this type of priority research paper.

Key Finding:

Opportunities exist for ACOA to make use of best practices developed across the Agency to improve its PAC function.

Case studies reveal several best practices that could enhance internal coordination and communication of PAC activities. The aquaculture and Atlantic Gateway case studies are examples of promising practice for both internal and external coordination. Communication mechanisms used by the Aquaculture Working Group are considered effective and instrumental in moving the aquaculture industry forward with a coordinated region-wide approach. Frequent informal communication mechanisms among ACOA HO, the New Brunswick regional office and external stakeholders were noted as particularly strong. The aquaculture external key informants also spoke of long-standing, positive relationships with their ACOA counterparts and understanding of their roles and responsibilities.

While several best practices were noted, there is little evidence, with the exception of the OBT approach, that they have been communicated formally across the Agency. Improved communication and implementation of best practices across regional offices is an opportunity for the Agency to make use of its own knowledge to improve practices.

6. Conclusions and Recommendations

This report presents the results of the evaluation of ACOA’s PAC function. The evaluation examined the relevance and performance of PAC from 2005-2006 to 2009-2010. The conclusions that were drawn from the evaluation findings and the proposed recommendations are presented below.

6.1. Relevance

The evaluation results indicate that PAC plays a legitimate and necessary role within ACOA and Atlantic Canada and is aligned with the Agency’s mandate and strategic outcome, as well as government-wide priorities and strategies. The results indicate that PAC effectively meets the needs of the minister, external stakeholders and senior management. However, the evaluation identified needs of Programs and internal stakeholders that were not being met consistently by PAC units. These findings support the conclusion that PAC is highly relevant and aligned with ACOA’s mandate and Government of Canada priorities and strategies. PAC is effective in meeting the needs of external stakeholders, the minister and senior management, but there are challenges in meeting internal stakeholder needs.

6.2. Performance

PAC’s decentralized structure is appropriate and considered essential for economic development. The evaluation highlighted the importance of policy research and analysis to Programs and ACOA’s advocacy and coordination efforts, as well as to senior management decision making. PAC also plays an important role in program renewal and the establishment of new programs within the Agency. These findings support the conclusion that the PAC model and activities are appropriate in achieving intended results.

The evaluation identified the need to clarify and communicate PAC roles and responsibilities in a number of areas. Clarity and awareness of PAC roles and responsibilities, and coordination and communication across PAC units, could reduce the risk of overlap and duplication of PAC activities, which is increased by the decentralized model even though this model is appropriate. The evaluation also identified an opportunity to further optimize the use of ACOA’s specialized knowledge and skills base. These findings support the conclusion that the performance of PAC could be strengthened by clarifying roles and responsibilities and improving coordination and communication across PAC units (including the role of Policy Network) and between PAC and Programs.

Policy research and analysis is deemed current, relevant and useful to decision making. The evaluation also determined that there is adequate capacity to deliver on PAC results. However, the absence of effective coordination and communication mechanisms for policy research has resulted in a lack of awareness of work undertaken across the Agency and has increased the risk of overlap and duplication of effort. These findings support the conclusion that policy research and analysis are fundamental activities for the PAC function. The relevance and usefulness of this work to PAC stakeholders could be enhanced through increased planning, coordination, communication and accessibility. This would also reduce the risk of overlap and duplication.

Though PAC is viewed as having a lead role in strategic planning and priority setting for the Agency, there is an opportunity to improve the strategic planning process. These findings support the conclusion that PAC’s contributions to priority setting, strategic planning and setting a strategic direction for the Agency could be enhanced.

Performance measurement for policy functions such as PAC is a common challenge among other federal departments and agencies. PAC’s inclusion in ACOA’s PAA brings with it specific central agency requirements for performance measurement and reporting. While a performance measurement strategy exists for PAC, evaluation results indicated that the lack of availability of performance measurement data limits the degree to which PAC can be managed, planned for, measured and reported. These findings support the conclusion that performance measurement of PAC is currently not sufficient to support the results-based management of the PAC function.

Based on the above conclusions, the following recommendations are made:

Recommendation 1 – Acknowledging ACOA’s decentralized structure, ACOA’s senior executives, led by the Senior VP, should:

Recommendation 2 – ACOA's senior executives should review the corporate strategic planning process to ensure it:

Recommendation 3 – Under the functional direction of the Senior VP, the DGs of PAC should review the performance measurement strategy for PAC to ensure:

For a visual depiction of the alignment of conclusions and recommendations, see Appendix E.

Appendix A – Program Activity Architecture Chart

Strategic Outcome: A competitive Atlantic Canadian economy

Program Activity: Enterprise Development

Program Sub-Activities:

Program Activity: Community Development

Program Sub-Activities:

Program Activity: Policy, Advocacy and Coordination

Program Sub-Activities:

Program Activity: Internal Services

Program Sub-Activities:

Source: 2011-2012 Report on Plans and Priorities

Appendix B – Summary of 2010 Atlantic Policy Research Initiative Evaluation

The purpose of the Evaluation of ACOA’s Atlantic Policy Research Initiative, conducted by the Evaluation Unit and Finance and Corporate Services, was to explore the core issues of relevance, performance and cost-effectiveness. Following are the key findings and recommendations of the evaluation.

Relevance/Alignment with Government Priorities

Performance

Efficiency and Economy

Conclusion: Relevance/Alignment with Government Priorities

APRI is relevant and meets a demonstrated need for policy research in Atlantic Canada. Engagement and research activities supported by the program are aligned with the Government of Canada’s priorities and ACOA’s areas of interest, as well as the Agency’s policy, advocacy and coordination functions.

Conclusion: Performance

ACOA plays a key role in producing economic policy research and related engagements in the Atlantic region. The Agency has been successful in meeting APRI intended outcomes. The initiative provides ACOA with the knowledge required to support policy development, advocacy and coordination efforts, and has built a reputation for excellence in policy research. One area identified for improvement would be to have the APRI management and staff work more closely with ACOA Programs and regions to identify research needs.

Conclusion: Efficiency and Economy

APRI is considered to be cost-effective and to provide value for money. The evaluation results demonstrate that APRI activities are effective and emphasize the need for further development, which could be achieved by increasing promotion to new researchers, enabling more inclusive processes for setting research priorities and selecting projects, and strengthening communications practices and performance measurement of the initiative.

Recommendations

This evaluation has identified opportunities for improvement, leading to the following recommendations to further the achievement of desired APRI outcomes.

1.Implement more inclusive processes in setting priorities and selecting projects.

2.Develop internal and external communications plans in order to promote APRI and disseminate knowledge.

3.Keep an organized account of data on O&M projects, as these are subject to accountability requirements, along with the G&C projects to strengthen performance measurement. The data recorded should systematically include the rationale behind the dissemination strategy and parties to whom the report was delivered, irrespective of the medium (mail, e-mail, web links). Data on project outcomes should also be recorded.

4.Foster the application of the following best practices identified in APRI delivery:

Appendix C – Evaluation Question Matrix

The following are the evaluation issues and their associated indicators and data sources/methodologies.

Relevance: extent to which the PAC program activity addresses a demonstrable need and is relevant to ACOA’s mandate, strategic objectives as well as government wide priorities and strategies

  1. Is there a legitimate and necessary role for ACOA’s PAC programming?
  1. To what extent are PAC activities aligned with i) ACOA’s mandate and strategic outcome, and ii) government-wide priorities/strategies?
  1. To what extent is the PAC program activity meeting the needs of key stakeholders (both internal and external)?

Performance: Effectiveness: the extent to which program objectives have been achieved within the context of expected results and outcomes.

  1. Is the current governance structure for the PAC program activity appropriate? Are there any challenges/barriers related to the current structure which may be impacting on the achievement of expected results?
  1. In what manner and to what extent has PAC contributed to setting the priorities and strategic direction for the Agency?
  1. In what manner and to what extent has PAC played a role in the development and implementation of ACOA programming?
  1. To what extent is the policy research and analysis undertaken within PAC considered accessible and relevant by decision makers?
  1. Is the coordination of activities related to PAC adequate and effective between each of the following: ACOA regional offices, ACOA head office Policy, ACOA head office Ottawa, ACOA Executive Committee and/or Minister and Other levels of government?
  1. To what extent are the activities being undertaken in HO and the regions consistent with the logic model and performance measurement strategy for PAC? Are the outputs produced by PAC appropriate to support the achievement of expected results (i.e. are they the right outputs)?

Performance: Efficiency/Economy: the extent to which PAC activities are undertaken in an affordable manner, taking into consideration the relationship between outputs and the resources to produce them; the extent to which resources allocated to the PAC program activity are well-utilized, taking into consideration alternative delivery mechanisms.

  1. To what extent does annual spending on PAC activities reflect planned spending? What processes contributed to the cost variances from planned spending?
  1. How does the ACOA PAC program activity compare to other regional development agencies or similar organizations in terms of resources, structure, and other delivery aspects?
  1. Does senior management have access to ongoing information on the performance of the ACOA PAC function? If not, what improvements are required to enhance the function’s performance measurement information?
  1. What opportunities exist to improve the efficiency/effectiveness of ACOA’s PAC-related activities and outputs?

Appendix D – Data Collection Methodology

Approach

The approach to the evaluation was developed using a combination of document and literature review and preliminary interviews.

Thirty preliminary interviews were conducted during November and December 2010 with ACOA staff from PAC, ED and CD across all regions, as well as Ottawa and HO. These interviews gave the project team a broad perspective of how PAC is delivered across the Agency and allowed them to assess data availability for measuring performance. The process also assisted in validating the evaluation issues reflected in Appendix C. 

An extensive document and literature review identified the challenges that exist in evaluating policy activities, and highlighted the fact that there is no standard approach for evaluating policy. Given the complexities surrounding the policy function and the difficulties often encountered in capturing data, and in establishing and measuring long-term impacts, the literature cautions against formal approaches to program evaluation.

To date, there has been little audit or evaluation work conducted related to policy. A search of the TBS database of approved audits and evaluations revealed that, with the exception of the WD’s 2009 PAC review, there have been no federal evaluations of policy activities undertaken in recent years that could be used as a reference in developing an evaluation methodology. To date there has been limited guidance provided by TBS on acceptable methods for evaluating policy activities,[See Footnote lviii] largely resulting from the fact that prior to the 2009 Policy on Evaluation, internal functions such as these were largely excluded from the scope of evaluations within the Government of Canada.

While there is no standardized approach to evaluating policy-related activities, the recommended approach was one that was formative in nature, that focused on design and delivery of the policy function in relation to program delivery.[See Footnote lix] This suggested focusing on the achievement of short- to medium-term objectives and goals rather than on the measurement of long-term outcomes, while taking into consideration the reactive nature of policy work where priorities and tasks can shift quickly due to changing political agendas. A key first step was to understand the function and context in which PAC operates in order to understand the factors that impact the achievement of results.

Based on these findings, the approach to the PAC evaluation involved multiple lines of inquiry, with a stronger emphasis on qualitative than quantitative methods to address areas where data availability was limited. The approach also contained a number of formative evaluation elements related to effectiveness and efficiency (i.e. emphasis on design and delivery). Given the gaps that existed in data related to policy activities, the approach targeted profiling the activities undertaken in PAC HO, Ottawa and the regions in order to establish a baseline that could be used for future comparison. This approach enabled findings obtained from various methodologies to be compared and cross-validated.

Taking into account the challenges and issues associated with evaluating policy, advocacy and coordination activities, the following lines of inquiry were identified for the PAC evaluation:

Preliminary Interviews

As discussed above, the Evaluation Unit performed a series of preliminary interviews during November and December 2010 with ACOA representatives from PAC, CD and ED. Interviews were conducted either in person or by telephone.

Table 4: Distribution of Preliminary Interviews

Region Number of Participants
Head Office 5
Nova Scotia 7
New Brunswick 6
Prince Edward Island 5
Newfoundland and Labrador   5
Ottawa 2
Total 30

The interviews served three purposes: (1) they provided the evaluation team with an understanding of how the PAC program activity is delivered across the regions, Ottawa and HO; (2) they identified issues affecting the achievement of targeted outcomes as a means of validating the evaluation issues identified; and (3) they allowed for the assessment of the availability of performance data to support the evaluation.

Document and Literature Review

A comprehensive document and literature review of relevant internal and external documents was undertaken and was used to:

The documentation and literature review proceeded in parallel with the other elements of the data collection phase in order to reflect initial needs for background information, to design data collection instruments, and to cross-validate or supplement information obtained from other lines of inquiry.

Analysis of Project Data

While project data only represents a small proportion of the total workload undertaken by PAC, an analysis of project data allowed for an in-depth look at the BDP G&C-funded PAC projects. Details related to these PAC projects over the period of 2005-2006 to 2009-2010 were extracted from ACOA’s project database (QAccess) and used to profile the types of PAC projects supported by ACOA. For O&M-funded activities, some regions have used informal methods (e.g. Excel spreadsheets) to track information. Project data was supplemented by resource data from ACOA’s financial and human resource systems. This analysis was used to identify and assess areas such as:

Findings from this analysis were used to prepare an overall profile of the PAC program activity and helped identify areas where data gaps exist. In cases where data gaps were identified, data analysis was supplemented by qualitative methods of information gathering.

Comparative Analysis

A comparative analysis of the ACOA PAC function was undertaken in order to profile some of the design and delivery mechanisms of policy (including advocacy and coordination) activities at the federal level. The analysis also looked to identify lessons learned and best practices that may assist with improving program delivery.

Due to the highly developed nature and organization of ACOA’s PAC function (as compared to other newly established RDAs), it was determined that greater value could be obtained by exploring the design and delivery structures of policy functions (including advocacy and coordination where applicable) in other large federal organizations with a mandate that is not limited to economic development. Research was conducted to identify federal departments that represented a mixture of design and delivery models (e.g. centralized vs. decentralized). Comparison organizations included were CED-Q, FedNor, the WD, the Public Health Agency of Canada, Canadian Heritage, Finance Canada, and a comparable international organization, Highlands and Islands Enterprise Development.

Data were collected through document review and key informant interviews (n = 9). Examples of areas of assessment included organizational structure, resource levels, key roles and responsibilities, governance, coordination mechanisms, and performance measures and indicators.

In addition to the comparison of PAC with other federal organizations’ policy functions, a comparison of PAC across ACOA regions, Ottawa and HO was also conducted. For the internal ACOA comparison, data analysis was conducted related to the allocation of resources and FTEs as reported in ACOA’s financial and human resource systems in order to compare ACOA's capacity to carry out PAC activities across HO, Ottawa and the regions. This should help identify regional differences in program delivery and provide a baseline for the Agency for future studies.

Interviews – ACOA Staff and Stakeholders

Forty-four telephone or in-person interviews were conducted with key informants to gather qualitative and quantitative data. They included a cross-section of internal ACOA PAC senior management (n = 11) and staff (n = 17) as well as external stakeholders from academia (n = 3) and other federal (n = 8) and provincial (n = 5) government departments or agencies.

The ACOA staff interviews were a cross-section of VPs, PAC program managers, directors and DGs at HO and each of the regions involved directly with the delivery of PAC in the Agency. These interviews were used to explore views on the Agency’s role, effectiveness and efficiency in supporting policies and programs that strengthen the Atlantic economy.

One interview guide was developed with a separate section targeted at senior management. The interview guide was semi-structured, allowing for effective probing of issues and was based on the evaluation issues and questions identified in the evaluation framework in Appendix C of this document.

The scheduling and conduct of the interviews involved:

Interviews lasted approximately one hour. All interviews were conducted in the preferred language of the interviewee. Interviewees’ comments have been kept strictly confidential and only aggregated findings and non-attributed comments have been reported.

Key informants among PAC partners and stakeholders were identified with the help of the ESC and PAC program staff. As was the case with the ACOA staff interviews, key informant interviews were used to explore views on the Agency’s role, effectiveness and efficiency in supporting policies and programs that strengthen the Atlantic economy, as well as to assess the dynamics between partners and stakeholders, particularly from a coordination perspective.

As recommended by ACOA, PAC partners and stakeholders included academia, federal and provincial government departments, industry associations, networks, etc.

One interview guide was developed; however, certain questions were not applicable to all stakeholders. Contact names suggested during the preliminary interviews were used as a starting point, and the list was e-mailed to regional PAC contacts for further input. Prospective subjects for these interviews were sent an introductory letter from ACOA by e-mail that described the purpose of the evaluation and encouraged participation by the partners and stakeholders. The evaluation team then contacted each of the subjects to schedule and conduct the interviews using the same protocol as with the ACOA staff key informant interviews described above.

Case Studies

Case studies represented a qualitative approach to demonstrating the impact of PAC activities, which was used to illustrate ACOA’s role and impact in advancing PAC priorities and progress in implementing sector and horizontal strategies that strengthen the Atlantic Canada economy by highlighting best practices or lessons learned.

Case studies served as a useful method for obtaining qualitative information on projects, particularly in cases where the availability of quantitative data was limited. Criteria used for selecting the PAC case studies were:

A total of six case studies were completed:

  1. Atlantic Gateway
  2. Atlantic Canada Energy Office
  3. Aerospace and defence
  4. Aquaculture
  5. Opportunity–based teams approach
  6. Atlantic Population Table

Case studies were developed based on document reviews and in-person or telephone interviews with 36 key informants, which included a cross-section of ACOA staff and management (n = 24) and external stakeholders (n =12). In order to reduce the burden on PAC staff, those interviewed for case studies were excluded from the key informant interviews (where possible).

Case study information was reported in aggregate and was validated through consultations with the main ACOA contact for each case. Case studies were critical in highlighting lessons learned and best practices related to all three elements of the PAC function (and their interrelationships), while profiling the life cycle of policy from research and analysis to advocacy and coordination efforts.

Focus Groups

Five focus groups were undertaken with ACOA CD and ED program management (n = 13) and staff (n = 18) in each of the four regions and at HO. The focus groups explored the degree to which PAC was meeting the needs of CD and ED related to the implementation of ACOA programming, setting Agency priorities, and communicating policy research and analysis. The focus groups also played a key role in profiling PAC activities in the Agency and identifying ways of improving program delivery.

PGF Consultants Inc. was engaged to facilitate the focus group sessions. The evaluation team worked with the consultant to develop a focus group approach and guide to ensure a standardized format was used for each session. ACOA HO and regions were solicited to obtain participant names.

PGF Consultants Inc. was responsible for consolidating the findings from each session into a summary report provided to the evaluation team. Results from the focus groups were reported in aggregate in the evaluation report to maintain participant anonymity.

Appendix E – Alignment of Policy, Advocacy and Coordination Conclusions and Recommendations

Conclusion 1:

PAC is highly relevant and aligned with ACOA’s mandate and Government of Canada priorities and strategies. PAC is effective in meeting the needs of external stakeholders, the minister and senior management, but there are challenges in meeting internal stakeholder needs.

Conclusion 2:

The PAC model and activities are appropriate in achieving intended results.

Conclusion 3:

The performance of PAC could be strengthened by clarifying roles and responsibilities and improving coordination and communication across PAC units (including the role of Policy Network) and between PAC and Programs.

Conclusion 4:

Policy research and analysis are fundamental activities for the PAC function. The relevance and usefulness of this work to PAC stakeholders could be enhanced through increased planning, coordination, communication and accessibility. This would also reduce the risk of overlap and duplication.

Recommendation 1: Acknowledging ACOA’s decentralized structure, ACOA’s senior executives, led by the Senior VP, should:

Conclusion 4:

Policy research and analysis are fundamental activities for the PAC function. The relevance and usefulness of this work to PAC stakeholders could be enhanced through increased planning, coordination, communication and accessibility. This would also reduce the risk of overlap and duplication.

Conclusion 5:

PAC’s contributions to priority setting, strategic planning and setting a strategic direction for the Agency could be enhanced.

Recommendation 2: ACOA's senior executives should review the corporate strategic planning process to ensure it:

Conclusion 6:

Performance measurement of PAC is currently not sufficient to support the results-based management of the PAC function.

Recommendation 3: Under the functional direction of the Senior VP, the DGs of PAC should review the performance measurement strategy for PAC to ensure:

Appendix F – Management Action Plan

DATE: March 22, 2012

PROJECT TITLE: Evaluation of the ACOA Policy, Advocacy and Coordination Program Activity

RESPONSIBILITY CENTRE: Senior Vice-President’s Office

RESPONSIBILITY CENTRE MANAGER: Senior Vice-President, David Slade

Conclusion:

PAC is highly relevant and aligned with ACOA’s mandate and Government of Canada priorities and strategies. PAC is effective in meeting the needs of external stakeholders, the minister and senior management, but there are challenges in meeting internal stakeholder needs.

Conclusion:

The PAC model and activities are appropriate in achieving intended results.

Conclusion:

The performance of PAC could be strengthened by clarifying roles and responsibilities and improving coordination and communication across PAC units (including the role of Policy Network) and between PAC and Programs.

Recommendation 1: Acknowledging ACOA’s decentralized structure, ACOA’s senior executives, led by the Senior VP, should:

Management Response: Concur with recommendation.

Given the scope of the evaluation (period of review ending March 31, 2010), management notes the following governance practices and activities that have been implemented since March 31, 2010, that help respond to Recommendation 1:

Planned Action 1: Under the direction of the Senior VP, the HO Policy team along with the regional offices and the Ottawa office, will define the roles and responsibilities for all PAC units in ACOA. This work will include:

This work will prioritize an Agency-wide corporate approach. The proposed planned action will be subject to Executive Committee review and recommendation to the President for approval.

Responsibility: Senior Vice-President

Target Date: March 2013

Planned Action 2: The Policy Network and DGs PAC meetings and conference calls will be held on a set timetable (i.e. every other week). The agenda for these meetings will be developed with the Agency’s priorities in mind, based on Executive Committee discussions, presidential direction and ministerial priorities as primary inputs.

Responsibility: DG Policy / PAC DGs

Target Date: April 2012

Conclusion:

Policy research and analysis are fundamental activities for the PAC function. The relevance and usefulness of this work to PAC stakeholders could be enhanced through increased planning, coordination, communication and accessibility. This would also reduce the risk of overlap and duplication. Conclusion:

PAC’s contributions to priority setting, strategic planning and setting a strategic direction for the Agency could be enhanced.

Recommendation 2: ACOA's senior executives should review the corporate strategic planning process to ensure it:

Management Response: Concur with recommendation

Planned Action 1: Under the direction of the Senior Vice-President, the DG of Policy at HO, in consultation with regional DGs PAC, will drive the renewal and revision of ACOA’s strategic planning process. The work will consist of the following:

A guiding principle behind this planned action will be a focus on corporate priority setting, with all ACOA offices contributing to corporate goals and priorities in a meaningful way.

Responsibility: Senior Vice-President

Target Date: September 2013

Conclusion:

Performance measurement of PAC is currently not sufficient to support the results-based management of the PAC function.

Recommendation 3: Under the functional direction of the Senior VP, the DGs of PAC should review the performance measurement strategy for PAC to ensure:

Management Response: Accept recommendation

Planned Action 1: With the assistance of the Agency’s performance measurement unit, and building on the Agency’s previous review of its PAC measurement approach with a recognized expert in the field of performance measurement, a review of PAC’s performance measurement strategy will be undertaken involving all PAC units. The review will:

Responsibility: Senior Vice-President

Target Date: September 2013

Appendix G – Reference notes

[i]Government Organization Act, Atlantic Canada,1987, Part 1, R.S., c G-5-7.

[ii] Treasury Board of Canada Secretariat. Policy on Evaluation. (Ottawa: Government of Canada, 2009). http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?section=text&id=15024

[iii] In order to properly contextualize findings, some references have been made to activities occurring outside the scope of this evaluation.

[iv] ACOA Evaluation Unit. Evaluation of the Atlantic Canada Opportunities Agency’s Atlantic Policy Research Initiative. (Moncton. Government of Canada, 2010). http://www.acoa-apeca.gc.ca/eng/Accountability/AuditsAndEvaluations/Pages/APRI.aspx

[v] The Federal Economic Development Agency for Southern Ontario and the Canadian Northern Economic Development Agency were excluded from the comparative analysis given that they were only established in 2009 and that their policy and programming has not yet reached a similar level of organizational development as at ACOA, WD, CED-Q and FedNor.

[vi] PAC management is defined as Executive (EX) level or with the title of director.

[vii] Key informants from the Enterprise Cape Breton Corporation – a Crown corporation that administers programs on behalf of ACOA on Cape Breton Island – are classified as external.

[viii] Mayne, John. “Addressing attribution through contribution analysis: Using performance measures sensibly.” Canadian Journal of Program Evaluation, 10, 1 (2001): 1-24.

[ix]Government Organization Act, Atlantic Canada 1987, Part 1, R.S., c G-5-7. http://laws-lois.justice.gc.ca/eng/acts/A-13.7/FullText.html

[x] ACOA. Supplementary Information, ACOA’s 5-Year Report to Parliament, 2003-2008. (Moncton, Government of Canada, 2008), p. 64-72.

[xi] In 2005, ACOA introduced the concept of champion files. These VP-led initiatives aim to ensure leadership, engagement, visibility, credibility, focus and an integrated, Agency-wide approach on key horizontal PAC files. While the evaluation did not find any formal documented evidence of the termination of the champion file model, key informants stated that this model is no longer used by the Agency. Rather, most champion files are Agency priority sectors, activities or advocacy files.

[xii] ACOA. Policy, Advocacy and Coordination Program Activity Evaluation. Preliminary Interview Results, 2010. (Moncton. Government of Canada, 2010).

[xiii] ACOA. ACOA Performance Measurement Strategy. (Moncton. Government of Canada, June 30, 2010), p. 27-28.

[xiv] Ibid, p. 28.

[xv] ACOA. Policy, Advocacy and Coordination Evaluation Terms of Reference, April 19 2011, p. 5.

[xvi] ACOA. ACOA Performance Measurement Strategy. (Moncton. Government of Canada, June 30, 2010), p.29.

[xvii] ACOA. Policy, Advocacy and Coordination Evaluation Terms of Reference, April 19 2011, p. 5.

[xviii] ACOA. ACOA Performance Measurement Strategy. (Moncton. Government of Canada, June 30, 2010), p. 30.

[xix] ACOA. Policy, Advocacy and Coordination Evaluation Terms of Reference, April 19 2011, p. 6.

[xx] ACOA. Policy Network Terms of Reference – Final Draft. (Moncton. Government of Canada, September 2011), p. 1.

[xxi] In order to provide a more accurate picture of PAC-specific expenditures, the cost of internal services, Treasury Board-funded costs and other statutory and miscellaneous expenditures have been excluded from subsequent analyses. These costs amount to $11.0 million over the five-year period from 2005-2006 to 2009-2010, or an average of $2.2 million a year.

[xxii] The groups are defined as follows: EC – economics and social science; CO – commerce officers; AS – administrative services; CR – clerical and regulatory; LS – library science; IS – information services; EX – executive group

[xxiii] Privy Council Office. The Eighteenth Annual Report to the Prime Minister on the Public Service of Canada, 2011. (Ottawa: Government of Canada, 2011). 

[xxiv] FedNor falls under the governance of Industry Canada.

[xxv] Supply and Services Canada. Regional Development in Canada. Prepared by Guy Beaumier Economics Division. (Ottawa: Government of Canada, Revised 27 October 1998). http://publications.gc.ca/collections/Collection-R/LoPBdP/CIR/8813-e.htm

[xxvi]Government Organization Act, Atlantic Canada,1987, Part I, R.S., c G-5-7. http://laws-lois.justice.gc.ca/eng/acts/A-13.7/FullText.html

[xxvii] Pestieau, Caroline. Evaluating Policy Research – Research Paper W/22, Work Network, (Ottawa: Canadian Policy Research Networks, December 2003), p.1.

[xxviii] ACOA. Evaluation of the Atlantic Canada Opportunities Agency’s Atlantic Policy Research Initiative: Final Report. (Moncton: Government of Canada, March 2010). http://www.acoa-apeca.gc.ca/eng/Accountability/AuditsAndEvaluations/Pages/APRI.aspx

[xxix] Ibid.

[xxx] The Next Phase of Canada’s Economic Action Plan, The Challenge

[xxxi]Treasury Board of Secretariat of Canada. Whole of Government Framework. http://www.tbs-sct.gc.ca/ppg-cpr/frame-cadre-eng.aspx

[xxxii] ACOA. Integrated Business Plan, 2010-2011. (Moncton: Government of Canada, 2010)

[xxxiii] Ibid, p.12.

[xxxiv] ACOA, Atlantic Policy Research Initiative Evaluation, (Moncton: Government of Canada, March 2010). http://www.acoa-apeca.gc.ca/eng/Accountability/AuditsAndEvaluations/Pages/APRI.aspx

[xxxv] Evidence of achievement has been extracted from six case studies completed as part of the PAC evaluation.

[xxxvi] Treasury Board Secretariat. Policy on Management, Resources and Results Structures. http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=18218&section=text#appA

[xxxvii] Institute on Governance. Public Sector Governance

[xxxviii] Policy Research Initiative. Capacity, Collaboration and Culture: The Future of the Policy Research Function in the Government of Canada, (Ottawa, Government of Canada, March 2009), p.5.

[xxxix] Canadian Policy Research Networks. The Future of Policy Capacity in Canada: Roundtable Report. May 2009.

[xl]Finance Alberta . Results Oriented Government. A Guide to Strategic Planning and Performance Measurement in the Alberta Government. Module 1:Strategic Planning Module. (Calgary: Government of Alberta). 

[xli] Privy Council Office. Report of the Expert Panel on Integrated Business and Human Resources Planning in the Federal Public Service. (Ottawa: Government of Canada, 2008). 

[xlii] Government of Canada. The Fifth Report of the Prime Minister’s Advisory Committee on the Public Service: A Public Service for Challenging Times. (Ottawa: Government of Canada, March 2011). 

[xliii] ACOA. ACOA Executive Planning Session. Article for ACOA’s Rendez-vous website.

[xliv] Policy Research Initiative. Capacity, Collaboration and Culture. The Future of the Policy Research Function in the Government of Canada. (Ottawa: Government of Canada, March 2009).

[xlv] As a result of the APRI evaluation, the Agency has recently put in place a Policy Research Working Group to address these issues; however, its implementation is outside the scope of the current evaluation. See Section 5.1 for more details.

[xlvi] ACOA. Conceptual Framework for Operational Planning. Policy, Advocacy and Coordination. EX Planning Session. (Nova Scotia, September 7-8, 2006).

[xlvii] ACOA QAccess and Operations and Maintenance Data. August 2011.

[xlviii] Western Economic Diversification Canada. Evaluation of PAC Functions. Audit and Evaluation Branch. July 2009.

[xlix] ACOA. Policy Network Terms of Reference – Final Draft. (Moncton: Government of Canada, September 2011), p. 1.

[l] Ibid, p. 1.

[li] ACOA, Report on a Change Management Strategy: Opportunity-Based Teams (Moncton: Government of Canada, 2010), p. 13.

[lii] SharePoint is a server-based Microsoft product that supports information management business processes and business intelligence needs.

[liii] It should be noted that the Federal Economic Development Agency for Southern Ontario and the Canadian Northern Economic Development Agency, which were created in 2009, include PAC in their program activity architecture. These were, however, excluded from the comparative analysis. See page 3 for further details.

[liv] Organization for Economic Collaboration and Development. “Factors that Hinder or Facilitate the Use of Indicator Systems.” Chapter 3 in Governing Regional Development Policy, the Use of Performance Indicators. 2009.

[lv] In this context, output tracking refers to the use of data systems or other means to store basic information (e.g. title, topic, description) related to outputs produced.

[lvi] While total budgets exclude CEAP amounts for each RDA, this exclusion was not done for FTE analysis given lack of information across RDAs.

[lvii] Organization for Economic Collaboration and Development. “Factors that Hinder or Facilitate the Use of Indicator Systems.” Chapter 3 in Governing Regional Development Policy, the Use of Performance Indicators. 2009.

[lviii] The Centre of Excellence for Evaluation has established a Policy Program Working Group to develop guidance for evaluating policy-related activities.

[lix] In this context, program delivery refers to how PAC is designed and delivered as a program activity within the PAA.

Page details

Date modified: