This evaluation provides management of the Atlantic Canada Opportunities Agency (ACOA) with systematic, neutral evidence on the relevance and performance of its Atlantic Policy Research Initiative. The results of this evaluation will be used to support ongoing program management and future direction of the programming.

The study was managed and implemented by ACOA’s Evaluation Unit. We thank the members of the evaluation working group and the Policy Research Working Group for their advice and support throughout the evaluation process. Their involvement and assistance has helped to ensure that the study and its recommendations are both relevant and useful.

We are also grateful to the many ACOA staff members, including members of the Evaluation Unit, clients and external key informants who provided their time and essential knowledge in support of this study.

Overall, these contributions were instrumental in building our understanding and identifying best practices and areas for improvement for ACOA’s Atlantic Policy Research Initiative.

Courtney Amo Director, Evaluation and Risk Directorate (Head of Evaluation) Atlantic Canada Opportunities Agency

Executive summary

Background

The Atlantic Policy Research Initiative (APRI), a pan-Atlantic program coordinated by ACOA’s Head Office (HO), was created in 2000. It is one of the main programming tools through which the Agency carries out its external policy research responsibilities. It contributes to each of the sub-program activities of the Policy, Advocacy and Coordination (PAC) program.

PAC programming identifies and responds to opportunities and challenges facing the Atlantic region’s economy. The branch offers intelligence, advice and analysis to support decision making that affects Atlantic Canada’s economic development.

The purpose of APRI is to enhance the capacity of the Agency to co-ordinate and plan pan-Atlantic federal activities contributing to economic growth in Atlantic Canada. APRI facilitates closer and more effective integration of the policy research process with all of the Agency’s strategic priorities. It also helps to support ACOA’s economic development mandate role by extending ACOA’s horizontal policy reach in Atlantic Canada by involving an expanding network of regional partners, including independent public policy organizations, government departments and agencies, universities, colleges, the economic community, business associations and independent researchers.

To achieve its objective, APRI uses both grants and contributions (G&Cs) and operations and maintenance (O&M) funding instruments. APRI G&Cs support proponent-initiated policy research, whereas the APRI O&M is used in support of ACOA-initiated research. As such, the G&C component respects the Government of Canada Transfer Payment Policy while APRI O&M funding respects the Government of Canada contracting policy. Key APRI activities include research studies and reports, support of policy or research collaborations through engagement activities, and initiatives that promote networking on policy-related issues. APRI funding assists in the coordination of policies, programs and partners to develop and foster collaborative approaches to economic development at ACOA.

Projects approved under APRI between 2009-2010 and 2013-2014 included G&C funding in the amount of $2.8 million and O&M funding in the amount of $0.435 million. APRI activities were supported annually by approximately 2.1 staff members or full-time equivalents.

The purpose of this evaluation was to assess the relevance and performance of the APRI program. This is the third evaluation of the program, with previous studies conducted in 2005 and 2010. The evaluation was undertaken to fulfill Government of Canada accountability requirements as well as the informational needs of ACOA’s senior management. It was implemented according to the terms of reference for the study, as approved by ACOA’s President in September 2014.

The evaluation methodology was calibrated to reflect the relatively low risk and materiality of this program. It included interviews with 20 key informants (internal staff and external proponents), a document and literature review of more than 50 documents, a compilation of four case studies of major funded projects, as well as administrative and financial analysis incorporating data from a variety of sources. Evaluation findings are based on a high level of convergence of multiple lines of evidence and are deemed reliable and valid within the context of the study limitations.

Conclusions

Relevance

The APRI program is relevant. There is an ongoing need for policy research funding to address critical gaps in knowledge about the future of economic development in Atlantic Canada. Engagement and research activities supported by the program are aligned with the mandate and priorities of the Government of Canada and of the Agency. APRI’s unique nature and ability to support broad policy research activities allows the program to complement rather than duplicate other ACOA/regional programs.

Performance – effectiveness

ACOA plays a key role in supporting the development of economic policy research and networking opportunities in the Atlantic region. APRI activities have been successful in supporting the immediate outcomes of the PAC branch.

Steps have been taken to improve communication and engagement since the 2010 APRI evaluation, including the establishment of the Policy Research Working Group (PRWG) and the development of research priorities. Opportunities exist to further enhance internal engagement on priorities and outputs and to improve internal and external communication. In particular, additional efforts to reach out to program areas and other interested parties to establish or validate research priorities, and to share information on completed policy research activities across ACOA, would enhance the effectiveness of this engagement.

Knowledge mobilization is a critical component in transferring information to decision-makers in an increasingly complex policy environment. There are opportunities to enhance the knowledge mobilization of outputs from APRI-funded projects with both internal and external audiences.

Performance – efficiency and economy

ACOA recognizes the importance of efficient planning and governance in the delivery of APRI and has made progress in terms of establishing the PRWG and encouraging collaborative relationships between key funding partners and ACOA. As one of the tools used by ACOA to deliver external policy research activities, APRI is an efficient and economical program. There are opportunities to further strengthen current operational efficiencies, including the enhancement of APRI’s operational processes and tools, and continued and enhanced implementation of the best practices identified in this evaluation.

As the APRI logic model and the performance measurement strategy (PMS) had not been updated since the 2010 evaluation, limited outcome level performance measurement data were collected. It is anticipated that the upcoming PAC logic model exercise will support APRI in better planning and monitoring the program’s results and indicators.

Recommendations

The three evaluation recommendations reflect discussion and advice from the members of the evaluation working group and the PRWG. Each of the recommendations aims to build upon progress made to program delivery, performance and efficiency since the previous evaluation, while ensuring that ACOA identifies and considers emerging programming needs on an ongoing basis.

In the spirit of continuous improvement, it is recommended that APRI management implement the following recommendations:

Build on recent strategic engagement efforts by seeking strategic input from ACOA Programs and other interested parties in establishing policy research priorities and sharing information about past and current policy research. (Internal Engagement)
Explore opportunities to further enhance the knowledge mobilization of outputs from APRI-funded projects with both internal and external audiences. (Knowledge Mobilization)
In tandem with current PAC efforts, calibrate the performance measurement approach to better inform the APRI program and Agency decision making by identifying, tracking and analyzing key indicators that focus on monitoring both program outputs and immediate outcomes. (Performance Measurement)

Management has agreed with this evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the evaluation’s recommendations, can be found in Appendix A.

1.0 Introduction

This report presents the findings, conclusions and recommendations of the evaluation of the Agency’s APRI. The evaluation reports on the relevance and performance of APRI based on the ACOA’s approved evaluation plan and the study’s approved terms of reference, which take into consideration the needs of senior management. It also fulfills accountability requirements set forth by the Treasury Board (TB) Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada.

PAC programming identifies and responds to opportunities and challenges facing the Atlantic region’s economy. PAC offers intelligence, advice and analysis to support decision making that affects Atlantic Canada’s economic development. In addition to its O&M funding (e.g. internal analysis, O&M contracts), ACOA has two dedicated G&C programs (APRI and the Business Development Program) that support the Agency’s objective of increasing policy research and capacity in Atlantic Canada.

An important part of the policy development model espoused by ACOA is the effectiveness with which results from research or engagement events are disseminated to a wider audience. This can occur at two levels – within ACOA (up to and including the minister) and through APRI proponents who disseminate the research to audiences outside ACOA. As noted above, APRI is the dedicated mechanism to conduct Atlantic-wide policy research. Research of provincial or local scope is also done in ACOA regional offices or through other programs, mainly the Business Development Program (BDP). Beyond this, policy development draws on many sources outside the unit, so much so that attributing the impact of different contributions to a policy is challenging.

This evaluation covered a five-year period, from 2009-2010 to 2013-2014. Background research was undertaken between June 2014 and November 2014. ACOA’s Evaluation Unit planned and conducted the evaluation. An evaluation working group, with representatives from ACOA’s APRI management and staff, provided advice and guidance throughout the evaluation.

The report is structured as follows: sections 1 and 2 provide an overview of the evaluation approach and a profile of the APRI program. Sections 3 to 5 present the evaluation findings by broad areas of relevance, performance-effectiveness, performance-efficiency and economy. Finally, section 6 presents the conclusions and recommendations.

Management has agreed with the evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the evaluation’s recommendations, can be found in Appendix A.

1.1 Evaluation overview

This evaluation provides timely, credible and neutral information on the relevance and performance of APRI to support decision making, continuous improvement and results-based management.

1.2 Evaluation design and methodology

This is the third evaluation of APRI. The initiative was previously evaluated in 2005 and 2010. While the 2010 evaluation found that APRI was relevant and was largely achieving its intended results, the report made four recommendations related to priority setting, communication, performance measurement and best practices.

This evaluation addresses five core issues that fall into two broad categories – relevance and performance – in accordance with the TB Policy on Evaluation.

Table 1 identifies the specific evaluation questions for each core issue as per the terms of reference prepared in consultation with the evaluation working group and approved by the president. A detailed evaluation framework, including core questions, judgment criteria and performance indicators can be found in Appendix B.

The evaluation used a mixed-methods research design involving multiple lines of evidence gathered through both quantitative and qualitative methods. In general, quantitative methods such as data analysis are relied upon when describing activities and outputs. Qualitative methods such as document reviews, interviews and case studies support the examination of short- and intermediate-term outcomes and impacts of these investments, as well as relevance issues.

Given that the program represents minimal risk and expenditures and that comprehensive evaluations were conducted in 2005 and 2010, a smaller-scale, less costly evaluation was designed. To answer the evaluation issues and questions, several methods were used for the APRI evaluation, including a document and literature review, project and administrative data analysis, interviews/focus groups and case studies.

Table 1: APRI Evaluation Questions by Core Issue

Issue	Question
Relevance
Issue 1: Continued Need for the Program
1.	What is the current situation in Atlantic Canada with respect to the need for economic policy research and engagement? To what extent are the needs of stakeholders being met? To what extent, and how, has APRI been responsive to existing and emerging policy research needs?
Issue 2: Alignment with Government Priorities
2	To what extent is APRI aligned with federal government and ACOA roles and priorities relating to economic policy research and engagement in Atlantic Canada?
Issue 3: Alignment with Federal Roles and Responsibilities
3	To what extent, and how, do other policy research funding programs or mechanisms complement, overlap or duplicate the objectives of APRI within ACOA and/or at the federal, provincial or local levels (e.g. academic, industry, etc.)?
Performance
Issue 4: Effectiveness
4.1	How well is APRI achieving its pan-Atlantic program objectives? Does APRI contribute to the immediate outcomes of PAC (see Section 2.2 for rationale for use of PAC logic model)? If so, to what extent and in what ways has APRI contributed to: Policy: well-informed policy decisions reflecting opportunities and challenges of the Atlantic region’s economy while considering enterprise and community development potential, including contributions to the development of strategies Advocacy: Atlantic enterprise and community development interests being considered in emerging and changing federal economic policies, programs and regulations Coordination: the coordination of partners in addressing the economic priorities of Atlantic Canada through a coherent approach to development, including outreach/engagement, advocacy and capacity building
4.2	In what ways has policy research been made available to decision-makers in a timely manner to act on economic issues?
4.3	To what extent, and in what ways, do facilitative activities (e.g. engagement, outreach, relationship building, exploration, consensus building) support the achievement of broader program objectives such as the promotion of forward thinking around key, evolving and sometimes higher risk issues?
4.4	To what extent, and in what ways, has there been reciprocal information sharing between the APRI program and both internal and external stakeholders (including program staff and regional offices), to support awareness of APRI and stakeholder objectives, engagement in terms of priority setting, and dissemination and use of outputs?
4.5	To what extent, and how, can social media and other technology be used to engage with stakeholders and support communication and dissemination of policy research findings and conclusions?
Issue 5: Efficiency and Economy
5.1	How effective are the mechanisms and structures within ACOA that facilitate policy research governance (e.g. planning, conducting, dissemination and use)? To what extent is the Agency making optimal use of the knowledge assets generated through policy research? What are the barriers to this optimization?
5.2	Is there a more cost-effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned? What can APRI learn from other economic policy funding mechanisms in other regional development agencies or similar organizations?
5.3	Is there appropriate performance measurement in place? If not, what steps are required to support the development and implementation of more effective performance measurement activities? If yes, is performance measurement information being used to inform senior management decisions?

Methods that rely on existing data

Document and Literature Review: A review of approximately 50 relevant internal (both Agency-wide and region specific) and external documents was undertaken. It included background and program documentation on APRI, including a selection of project files. Documents on related federal policies and strategies, as well as documentation on similar activities in other jurisdictions (including those implementing alternative modes of delivery), were also reviewed for possible lessons learned/best practices that may assist with improving program delivery (especially with regards to efficiency).
Project and Administrative Data Analysis: An analysis of G&C project data from ACOA’s QAccess database allowed for an in-depth look at APRI grant and contribution projects, including the types of activities funded by sector, research area and other elements. Project data was supplemented by data from ACOA’s GX financial system where possible. Project qualitative reviews (where available) were analyzed. Performance data on the program’s O&M projects, compiled by program staff for the evaluation, were also analyzed.

Methods that involve the collection of new data

Interviews/Focus Groups: Key informants (n = 20) were interviewed in person (when possible) or by teleconference and included a cross section of internal ACOA management and staff (n = 12), as well as external stakeholders (n = 8). Interviews explored lessons learned and best practices in the achievement of desired outcomes. A focus group with ACOA policy experts – the PRWG – was also undertaken.
Case Studies: A small thematically representative and purposeful sample (n = 4) of case studies provided an understanding of ACOA’s contributions to the achievement of immediate PAC outcomes. Each case study involved a review of project documents, including the project summary form (PSF), the analysis of QAccess data, the completion of a questionnaire by program staff, and interviews with project proponents. A cross-case analysis was completed to draw out findings and lessons learned across cases. Case studies included the following projects: functional economic regions (2014), bioenergy (2013), small and medium-sized enterprise (SME) financing (2012) and global value chains (2011).

Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included: systematic compilation, review and synthesis of data to illustrate key findings; analysis of quantitative data from databases; thematic analysis of qualitative data; and comparative analysis of data from different sources to validate summary findings.

1.3 Evaluation strengths, limitations and mitigating measures

In accordance with TBS evaluation standards (including acceptable approaches to calibration), the evaluation design and implementation are deemed appropriate for the objectives of the study. A key strength of the evaluation design was the consideration of the broader program theory, supported by APRI and PAC logic models.[i] Another important strength was the participatory approach to stakeholder engagement, including the use of an evaluation working group whose members supported the development and implementation of particular methods. The evaluation team also facilitated a discussion of the evaluation scope and later of preliminary findings and conclusions with the evaluation working group, as well as the broader PRWG. This evaluation was also able to build on the approaches and knowledge gained from the two previous evaluations of APRI (2005, 2010) as well as from the 2012 evaluation of PAC.

Most evaluations face constraints that may have implications for the validity and reliability of findings and conclusions. Table 2 outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings could be used with confidence to guide program planning and decision making.

Table 2: Evaluation Limitations, Impact and Mitigation Strategies

Limitations	Impact	Mitigating Strategies
Complex Program Context Attribution vs contribution.	Due to the complexity of addressing needs for policy research in Atlantic Canada, programming results can be influenced by a multitude of factors.	PAC logic model was used to explore APRI contributions to PAC key outcomes related to policy, advocacy and coordination functions.
Key Informant Interviews Interviews retrospective in nature and small number conducted in line with calibrated approach.	Interviews retrospective in nature, providing recent perspective on past events can impact validity of assessing activities or results relating to improvements in the program area.	Triangulation of other lines of evidence to substantiate or provide further information on data received in interviews: Document review provides corporate context and knowledge. Advice from the evaluation working group and the PRWG supported the interview process.
Limited availability of detailed financial data linking expenditures on other policy research at ACOA (O&M, BDP).	Limited ability to assess efficiency and economy.	Use of other data collection methods assisted in assessing economy and efficiency. Advice from the evaluation working group and the PRWG supported the data analysis process.
Limitations in performance data: Few benchmarks, baselines and targets were available. Output data stronger than outcome data.	While there was some performance measurement information available, in many cases the assessment of outcome achievement was difficult. Outcome measures, being qualitative in nature, were less measurable than output and activity measures, resulting in limited ability at times to assess evidence of achievement of outcomes.	Performance data was used to the fullest extent and provided indications of success in achieving some outcomes. Where information was lacking, triangulation of evidence from a document and literature review, document review, survey and key informants helped to validate findings and provide additional evidence of outcome achievement.

The diversity of methods helped compensate for the inherent limitations of each data source and helped mitigate the overall study challenges. Multiple lines of evidence gathered through a mix of qualitative and quantitative methods allowed for triangulation – meaning convergence of results across methods. The approach also allowed for complementarity of findings so that a better understanding of results was gained by exploring different facets of complex issues associated with policy research and knowledge mobilization.

Given these mitigation measures, the evaluation team is of the opinion that the limitations of the study were adequately addressed and the results of the APRI evaluation are deemed to be reliable and valid.

2.0 Profile of the Atlantic Policy Research Initiative

2.1 Context

Policy, advocacy and coordination

Within the Agency’s program alignment architecture (PAA – Appendix C), PAC is one of four program areas that support the Agency’s overall strategic outcome of developing “a competitive Atlantic Canadian economy.” PAC programming focuses on providing intelligence, analysis and advice to inform and support decision making. The function supports advocacy efforts to influence national policies and programs affecting Atlantic Canada.

The overall roles of each PAC sub-program are as follows:

Policy: Develop ACOA’s economic development policies and sectoral strategies to improve performance and realize opportunities.
Advocacy: Ensure that Atlantic Canada’s interests are recognized in the policies and programs developed by other federal departments.
Coordination: Work with other federal and provincial governments, institutions and the private sector to develop joint strategies and ensure a coherent approach.

Atlantic Policy Research Initiative

Established in 2000, the purpose of APRI is to enhance the capacity of the Agency to coordinate and plan pan-Atlantic federal activities contributing to economic growth in Atlantic Canada. Notably, APRI activities (through the use of G&C and O&M funding tools) also fit within the broader picture of policy research within ACOA, including supporting program activities within PAC, Enterprise Development and Community Development.

The objectives of APRI are:

to generate and disseminate knowledge that enhances policy knowledge, taking into account regional economic opportunities and challenges;
to support and influence the policy research priorities that impact the Atlantic region, particularly those directly related to economic development; and
to establish and promote networks among stakeholders.

The reach of APRI programming includes various organizations such as economic, business and industry associations, post-secondary educational institutions, research institutions and organizations, independent researchers, and provincial and municipal governments.

Projects and recipients selected for assistance include activities that:

have an Atlantic Canada scope (i.e. impact more than one Atlantic province);
contribute to the program’s objectives, ACOA’s strategic outcomes of Enterprise Development, Community Development and PAC, and support various themes, including business climate, competitiveness and growth, emerging sectors, human capital and social economy; and
are conducted in conjunction with at least one or more public- or private-sector partners or participants.

The maximum federal amount payable to any one project under APRI shall be $300,000 and the maximum amount payable to any one recipient over a five-year period shall be $1 million.

2.2 Program theory

A program’s theory serves to communicate the assumed causal connections between program elements. This theory is often presented visually in the form of a logic model, identifying the reach, activities and immediate and long-term (or strategic) outcomes.

During the planning phase, the working group explored options for using APRI or the broader PAC logic model to examine outcome achievements. The evaluation team conducted an exercise to compare the cross-cutting themes of the two models (see Appendix D). It was determined that the APRI logic model (currently under review) would assist in situating theory and context while the PAC logic model would be used to assess broader policy outcomes and how APRI feeds into the achievement of these expected outcomes.

As reflected in the 2010 APRI logic model, the overall program theory for APRI is based on the assumption that the target users are engaged in a way that encourages their meaningful participation and a mutual understanding of the context of economic policy research in Atlantic Canada. This engagement, and the funding and dissemination of policy research, should lead to improved knowledge and the ability of both internal and external stakeholders to facilitate well-informed policy decisions that reflect the opportunities and challenges of the Atlantic Region’s economy.

The key result for the APRI program is policies and programs that strengthen the Atlantic economy. Given the nature of policy research and the time required to realize long-term outcomes, the focus in the current evaluation was on the contribution of APRI to the outcomes of PAC.The PAC logic model (Figure 1) represents the expected outcomes of the three sub-programs: Policy, Advocacy and Coordination.

Figure 1: PAC Logic Model (2013)

Activities and Results	Policy	Advocacy	Coordination
Activities/Outputs	Analysis and focused research of policy issues, trends, challenges, opportunities, best practices and emerging approaches related to regional economic development. Macroeconomic, microeconomic and fiscal analyses. Engagement of research partners and stakeholders on Atlantic regional economic development issues and priorities.	Increased awareness and capacity-building in areas of strategic industrial interest, such as aerospace and defence. Involvement in the federal policy-making process.	Federal-provincial initiatives (number and type of formal and informal initiatives). Coordinated and shared initiatives with other federal departments via the federal regional councils. Partnership and network mechanisms that demonstrate a strong federal presence with regional and national stakeholders.
Sub-program Expected Results/Key Outcomes	Well-informed policy decisions reflecting opportunities and challenges of the Atlantic region’s economy while considering enterprise and community development potential.	Atlantic enterprise and community development interests are reflected in emerging and changing federal economic policies, programs and regulations.	Coordination of partners in addressing the economic priorities of Atlantic Canada through a coherent approach to development.
Program Expected Results/Key Outcomes	Policies and programs that strengthen the Atlantic economy
Strategic Outcome	A competitive Atlantic Canadian economy

Source: ACOA 2013-2014 Logic Models for ACOA programs and sub-programs, February 2013.

2.3 Program accountability and governance

ACOA’s executive committee (ExCom) is the highest internal governing forum that supports the President in the development of the Agency’s policies and programs. Members include ACOA’s vice-presidents and related senior executives, as identified by the president to ensure the appropriate representation of expertise and experience.

Governance and management of the PAC program is shared among HO, ACOA’s regional offices and Ottawa. The PAC structure is decentralized in that each regional office is equipped with policy capacity and each region has senior management responsible for policy who report to the regional vice-president.

Director general PAC committee

During the scope of the evaluation, the Director General (DG) PAC Committee assumed the responsibility previously held by the Agency’s Policy Network. The role of the DG PAC Committee is to provide the ExCom and individual vice-presidents with strategic and timely advice and identification of policy issues facing the region and ACOA. The committee oversees ACOA’s policy, coordination and advocacy work using an integrated team approach to share information, initiate and guide policy activities, and engage Agency resources in all regions as well as at HO on corporate policy priorities and strategic planning. The committee is chaired by the DG Policy at HO, and includes other DGs and Directors responsible for PAC at HO and regional offices. [ii]

Policy research working group

The purpose and mandate of the Agency’s PRWG is to support the DG PAC in its deliberations on key policy issues impacting the Atlantic economy and to ensure that the body of knowledge to support effective strategic decision making is available. The creation of this committee in 2013 was a deliberate approach to advancing collaboration within the PAC function as well as sharing information throughout the Agency.

The PRWG is a coordinating body for policy research activities at ACOA. The role of the working group is to coordinate and maximize the value of research, analysis and policy work funded by and developed within ACOA. Members of the working group are designated by the DG PAC Committee. The PRWG is chaired by the Manager, Policy Research and Information Services at HO. The Director, Strategic Policy Development sits on both the PRWG and DG PAC Committee and provides regular reports and feedback to both groups.[iii]

APRI management

APRI is a contribution program managed by a small HO team that reports to the DG of Policy. The team consists of the Director of Strategic Policy Development, the Manager of Policy Research and Information Services, the APRI Program Officer and the Administrator.

Project approval

Signing authority for APRI projects resides with the DG of Policy (HO), who can approve, reject, amend or withdraw a contribution agreement. The authority to disburse funds under these agreements is exercised in accordance with the Agency’s financial signing authority delegation instrument.

2.4 Expenditure profile

To achieve its objective, APRI uses both G&C and O&M funding instruments. APRI G&C supports proponent-initiated policy research, whereas O&M funding is used in support of ACOA-initiated research. As such, the G&C component respects the Government of Canada Transfer Payment Policy, while O&M funding respects the Government of Canada Contracting Policy. Two complementary funding streams are used by APRI. O&M funding is used primarily by APRI management to provide advice to ACOA by engaging the services of external policy experts. The G&C funding instrument, however, differs in that its intended reach and impacts extend beyond ACOA into the broader policy research community.

2.4.1 Project profile

G&C project data and O&M financial data were generated based on the project/contract approval dates. Differences in the amounts of annual funding across fiscal years can be attributed to: projects being approved in one fiscal year and funds expended in following years; federal initiatives that may have impacted the level of human resources available to support APRI; and the time it may take to get certain projects off the ground. The scope of the evaluation did not include O&M outside of APRI per a decision taken by the evaluation working group, in line with a calibrated approach.

Grants and contributions projects

The types of G&C projects funded under APRI are non-commercial and non-repayable in nature. Table 3 provides a summary of the G&C projects approved and funded by APRI. Of the 39 projects, 78 per cent of funding supported studies, while 22 per cent assisted networking projects (e.g., conferences, forums, summits supporting Atlantic Canadian research).

According to QAccess data, the 39 G&C APRI projects initiated from 2009-2010 to 2013-2014 represented a total of nearly $2.8 million in ACOA approved funding, or an annual average of $559,258.

Table 3: Approved G&C Projects between 2009-2010 and 2013-2014

Fiscal Year	Networking Projects	Studies	Total Number of Projects	Total Approved ACOA Assistance ($)
2009-2010	6	8	14	692,439
2010-2011	2	2	4	354,743
2011-2012	1	6	7	717,644
2012-2013	4	2	6	210,560
2013-2014	2	6	8	820,905
Total Projects	15	24	39	2,796,291
Total Approved	$620,866	$2,175,425	$2,796,291

Source: ACOA’s QAccess database, May 2014.

Operations and maintenance projects

The O&M funding breakdown, summarized in Table 4, is similar to that of G&C assistance in that 79 per cent of the funding supported studies, 19 per cent assisted networking activities and 2 per cent represented a combination of both networking and study activities. According to APRI management data, the 26 O&M projects initiated from 2009-2010 to 2013-2014 represented a total of nearly $0.435 million in ACOA approved/contracted funding, or an annual average amount of $86,964.

Table 4: Approved O&M Projects between 2009-2010 and 2013-2014*

Fiscal Year	Networking Projects	Networking and Studies	Studies	Total Number of Projects	Total ACOA Contract Amount ($)
2009-2010	4	1	4	9	143,174
2010-2011	2	1	5	8	90,022
2011-2012	0	0	4	4	65,649
2012-2013	1	0	4	5	135,975
2013-2014	0	0	0	0	0
Total Projects	7	2	17	26	434,819
Total Approved	$81,610	$9,551	$343,658	$434,819

Source: ACOA’s APRI Management, September 2014.

* No new projects were approved in fiscal year 2013-2014, however, funding from previous years’ projects was expended and project management efforts were required (e.g. Survey of Innovation and Business Strategy).

Grants and contributions activities

The activities supported under APRI G&C include: stakeholder-initiated external research; round tables focused on various policy issues and opportunity areas; and policy research conferences.

Examples of G&C project activities captured in the following table include a university conducting research on the value of e-business to productivity and growth; a research centre studying opportunities and challenges for Atlantic Canadian business trade with Asia; and a professional association offering a summit on the ocean and innovation as a source of economic prosperity.

Table 5 is a breakdown of the 39 G&C projects, initiated by proponents, funded during the scope of the evaluation. The research developed through these G&C projects and the related intellectual property (IP) is owned by the proponent. The average project size, in terms of ACOA assistance, was $72,000. The types of projects were analyzed by the evaluation team and verified by the APRI team based on the project descriptions found in QAccess.

Table 5: G&C Project Types between 2009-2010 and 2013-2014

Research and Networking Activities	Number of Projects Approved	% of Total Projects	Approved ACOA Assistance ($)	% of Overall ACOA Assistance
Economic Policy	9	23	960,090	34
Demographics	8	20	297,287	11
Innovation/Commercialization/Technology	7	18	358,480	13
Single-Sector Focus	7	18	480,740	17
Trade	5	13	656,786	23
Public Administration	3	8	42,908	2
Financial Administration	0	0	0	0
Total	39	100	2,796,291	100

Source: ACOA's QAccess database, May 2014.

Operations and maintenance activities

APRI O&M is used to fund ACOA-initiated research in accordance with Government of Canada contracting policy. Table 6 breaks down the 26 O&M projects, initiated by ACOA, funded during the scope of the evaluation. Unlike the G&C projects, research and the IP generated through these O&M projects is owned by ACOA since it is conducted as a service contract for the Agency. Some examples of O&M activities captured in the following table include: collaboration with other federal departments such as Industry Canada and Statistics Canada to conduct a survey to assess financing and growth of SMEs, and a research consultant contracted to assess biofuels in Atlantic Canada. The average project size in terms of ACOA assistance was $17,000. The types of projects were analyzed by the evaluation team and verified by the APRI team based on the project descriptions found in the O&M project data spreadsheets.

Table 6: O&M Project Types between 2009-2010 and 2013-2014

Research and Networking Activities	Number of Projects Approved	% of Projects	Approved ACOA Assistance ($)	% of Overall ACOA Assistance
Economic Policy	9	35	117,834	27
Demographics	0	0	0	0
Innovation/Commercialization/Technology	9	35	176,076	41
Single-Sector Focus	6	23	117,416	27
Trade	0	0	0	0
Public Administration	0	0	0	0
Financial Administration	2	7	23,493	5
Total	26	100	434,819	100

Source: ACOA’s APRI Management, September 2014.

3.0 Findings: relevance

The relevance of APRI was assessed by examining the continued need for the program and the alignment between the program, Government of Canada and Agency priorities, and federal roles and responsibilities. Overall, this evaluation found that APRI remains relevant. The needs expected to be met by the program exist to at least the same extent as they did five years ago. APRI activities are aligned with ACOA’s strategic outcome and the ACOA Act. They are consistent with the Agency’s roles and responsibilities and broader federal priorities related to economic development. APRI programming complements rather than duplicates other programming being delivered by ACOA and other organizations.

3.1 Continued need for the programming

Evaluation Question	Judgment Criteria	Key Findings
What is the current situation in Atlantic Canada with respect to the need for economic policy research and engagement?	The needs that the programming is expected to meet are still present to at least the same degree as they were five years ago.	#1: There continues to be a need for policy research to support stronger economic growth in Atlantic Canada. #2 There are limited opportunities for researchers to access economic development policy research funding in Atlantic Canada.
To what extent are the needs of stakeholders being met? To what extent, and how, has APRI been responsive to existing and emerging policy research needs?	As appropriate, current and planned program activities proactively address changes in context. Evidence of and views on activities and outcomes related to changes in the scientific, social and/or policy context in which the program operates. Evidence of and views on best practices and lessons learned that are applicable to current regulatory initiatives.	#3: APRI programming is responsive to existing and emerging policy research needs.

Evaluation Question

Judgment Criteria

Key Findings

What is the current situation in Atlantic Canada with respect to the need for economic policy research and engagement?

The needs that the programming is expected to meet are still present to at least the same degree as they were five years ago.

#1: There continues to be a need for policy research to support stronger economic growth in Atlantic Canada.

#2 There are limited opportunities for researchers to access economic development policy research funding in Atlantic Canada.

To what extent are the needs of stakeholders being met? To what extent, and how, has APRI been responsive to existing and emerging policy research needs?

As appropriate, current and planned program activities proactively address changes in context.

Evidence of and views on activities and outcomes related to changes in the scientific, social and/or policy context in which the program operates.

Evidence of and views on best practices and lessons learned that are applicable to current regulatory initiatives.

#3: APRI programming is responsive to existing and emerging policy research needs.

Finding #1: There continues to be a need for policy research to support stronger economic growth in Atlantic Canada.

APRI programming needs exist to at least the same extent as what was reported in the previous evaluations, as demonstrated through various lines of evidence. The document and literature review indicated that the economy of Atlantic Canada continues to lag behind other parts of the country. Closing the labour productivity gap is fundamental to Atlantic Canada’s economic prosperity.[iv] Some of the factors contributing to this gap include population changes and high unemployment in rural areas. Demographic changes such as declining population, mobility, aging of the population and immigration have a direct bearing on labour force development, availability and activity. For rural communities, the loss of population and labour to larger centres constricts the capacity for growth, development, expansion and innovation.

Internal and external key informants indicated that there is a critical need for continuing policy research in Atlantic Canada that addresses a number of significant gaps in knowledge about the future of economic development (e.g. labour market availability due to changing demographics and varying impacts of economic development on urban/rural communities). APRI addresses some of these knowledge gaps by supporting policy research organizations that undertake innovative and cutting-edge research in these areas. Key informant interviews with ACOA senior management show the value they place on quality evidence-based policy research to support policy decision making within ACOA.

Government and private-sector think tanks have underlined the importance of good evidence-based policy development practices, incorporating sound policy research and analyses. The federal publication Capacity, Collaboration and Culture – The Future of the Policy Research Function in the Government of Canada (2009), by the Government of Canada’s Policy Research Initiative, states: “Policy research helps the public service to understand and address current and emerging policy issues by providing impartial evidence-based research that can inform the policy development process.” This federal report further emphasized the need for region-specific analyses: “The national treatment of issues in the future will reflect less of a central Canada perspective in favour of regional perspectives, participation and autonomy.”

The evaluation team reviewed documentation and/or spoke with representatives from other federal RDAs such as Western Economic Diversification Canada, Canadian Economic Development for Quebec Regions (CED) and the Federal Economic Development Agency for Southern Ontario. While each organization has a policy research capacity, the ACOA APRI funding model was seen as a best practice by some RDA key informants as it provides consistent access to financial resources to conduct policy research activities.

Finding #2: There are limited opportunities for researchers to access economic development policy research funding in Atlantic Canada.

Internal and external key informants indicated that there are relatively few sources of funding available in Atlantic Canada for policy research activities to support economic policy development.This concern is particularly true for funding sources that have the flexibility to address knowledge gaps with “pan-Atlantic” perspective to support policy development needs. The funding sources that are available through provincial governments, academic institutions or the private sector are typically earmarked to address specific provincial or local needs.

The case study projects researched for this evaluation would not have happened in Atlantic Canada without the significant funding contribution of the APRI program. External proponents for the three G&C project case studies indicated that they would not have been able to undertake these projects without the development support of APRI program staff and the funding provided through ACOA. The proponent for the fourth case study, supported through O&M funding, indicated that research at the Atlantic level would not have taken place without APRI assistance.

Finding #3: APRI programming is responsive to existing and emerging policy research needs.

According to internal key informants, the planning of APRI program activities takes into consideration the current environment in which stakeholders/researchers must operate, including limitations surrounding access to capital, human resources and research expertise. In addition, the APRI team has unique and ongoing insight into the emerging issues impacting economic policy research in Atlantic Canada. For example, from the results of the SME financing study – funded through APRI O&M – analysis showed that Atlantic SME owners were older than those elsewhere in Canada, highlighting a need for succession planning within the Atlantic SME business community. The Agency’s PRWG then identified this as an area where additional policy research is needed.

The APRI team has generated a list of best practices associated with their projects. They have noted that APRI helps identify and support study topics that otherwise would not be funded and those studies have resulted in knowledge that likely would not have been captured. The work of the APRI team, as well as the recent implementation of the PRWG to foster regular communication with external stakeholders, regional policy experts and other potential funding partners has allowed them to be better attuned to the policy research environment in Atlantic Canada.

3.2 Alignment with government priorities

Evaluation Questions	Judgment Criteria	Key Findings
To what extent is APRI aligned with federal government and ACOA roles and priorities relating to economic policy research and engagement in Atlantic Canada?	There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome. The alignment is recognized, communicated and/or made explicit.	#4: Economic development in Atlantic Canada, and policy research to support it, is a priority for the Government of Canada ACOA.

Finding #4: Economic development in Atlantic Canada, and policy research to support it, is a priority for the Government of Canada and ACOA.

The purpose of APRI is to enhance the capacity of the Agency to coordinate and plan pan-Atlantic federal activities contributing to economic growth in Atlantic Canada. A priority for the Government of Canada continues to be economic growth. This priority is stated in a number of key federal planning documents, including the Speech from the Throne in October 2013, which stated that “creating jobs and securing economic growth is and will remain our Government’s top priority,” and the Budget Speech in February 2014, which stated that “creating jobs and opportunities remains our Government’s top priority.”

Over the past five years, policy research generally (and often APRI specifically) has been identified each year as an organizational priority in ACOA’s Report on Plans and Priorities (RPP). The RPP for 2014-2015 states that “through its policy research, analysis and engagement activities, which provide the foundation for the development of strong policies and programs, ACOA will continue to contribute to a competitive Atlantic Canadian economy and to support advocacy and coordination activities. It will conduct analysis and focused research of policy issues and trends in key areas reflecting opportunities and challenges related to the Atlantic region’s economy.” Also, projects funded by APRI are in line with ACOA priority areas for investment identified in previous and current RPPs, including changing demographics, international trade, innovation and strategic sectors.

In ACOA’s PAA, PAC is one of four programs supporting ACOA’s strategic outcome of “a competitive Atlantic Canadian economy.” APRI, through its contributions to PAC, ultimately supports the achievement of the Agency’s strategic outcome.

3.3 Alignment with federal roles and responsibilities

Evaluation Questions	Judgment Criteria	Key Findings
To what extent, and how, do other policy research funding programs or mechanisms complement, overlap or duplicate the objectives of APRI within ACOA (Enterprise Development and Community Development) and/or at the federal, provincial or local levels (e.g. academic, industry, etc.)?	ACOA is mandated by law to fulfill the role. Other jurisdictions administer such programming through the federal government. The APRI program complements rather than duplicates or overlaps other federal/provincial programming. Needs are coordinated with alternative services or funding programs within ACOA (extent of complementarity, duplication or overlap). Opportunities for improved coordination are being acted upon.	#5: There is a leadership role for the Government of Canada and ACOA in policy research related to economic development in Atlantic Canada. #6: APRI activities complement rather than duplicate other ACOA programming as well as other economic policy research programming in the region.

Evaluation Questions

Judgment Criteria

Key Findings

To what extent, and how, do other policy research funding programs or mechanisms complement, overlap or duplicate the objectives of APRI within ACOA (Enterprise Development and Community Development) and/or at the federal, provincial or local levels (e.g. academic, industry, etc.)?

ACOA is mandated by law to fulfill the role. Other jurisdictions administer such programming through the federal government.

The APRI program complements rather than duplicates or overlaps other federal/provincial programming.

Needs are coordinated with alternative services or funding programs within ACOA (extent of complementarity, duplication or overlap).

Opportunities for improved coordination are being acted upon.

#5: There is a leadership role for the Government of Canada and ACOA in policy research related to economic development in Atlantic Canada.

#6: APRI activities complement rather than duplicate other ACOA programming as well as other economic policy research programming in the region.

Finding #5: There is a leadership role for the Government of Canada and ACOA in policy research related to economic development in Atlantic Canada.

The authority for policy activities related to economic development in Atlantic Canada was established under the Atlantic Canada Opportunities Agency Act. It states that “the minister shall coordinate the policies and programs of the Government of Canada in relation to opportunities for economic development in Atlantic Canada.” As the federal department responsible for economic development in Atlantic Canada, ACOA has a role to play in undertaking policy research and analyses to establish a stronger Atlantic Canadian economy. By facilitating and funding economic policy research activities in Atlantic Canada, APRI activities contribute to ACOA’s mandate to “increase opportunities for economic development in Atlantic Canada and, more particularly, enhance the growth of earned incomes and employment opportunities in that region.”

Other federal government agencies have a role in fulfilling policy research mandates for economic regions outside of Atlantic Canada, including: Western Economic Diversification Canada, Canadian Economic Development for Quebec Regions and the Federal Economic Development Agency for Southern Ontario. Interviews with contacts in RDAs confirmed that they undertake evidence-based policy research to support decision making in their regional economic development organizations.

Finding #6: APRI activities complement rather than duplicate other ACOA programming as well as other economic policy research programming in the region.

During the 2010 APRI evaluation, a comprehensive assessment of other ACOA programming was completed, including a comparison of the terms and conditions of APRI and the BDP. It was reported that “though there may be some overlap between the type of research that can be conducted under APRI and the type of research conducted under the BDP, it is clear that APRI policy research objectives are broader than what is contained in the BDP, both in terms of the type of research that can be conducted (not solely focused on a likely impact for the development of SMEs) and in terms of the round tables or engagements. The policy research conducted under APRI is focused on broad contextual policy issues that influence economic development rather than on SMEs in particular.” The current 2015 evaluation, with its calibrated approach, took into consideration the previous findings as well as he information provided by current key informants.

APRI (external policy research) activities appear to complement ACOA programming as well as other economic policy research programming in the region. Internal key informants see APRI as a unique tool that complements other ACOA programs with its support of broad policy research issues. External key informants have indicated that there were no areas of potential overlap or duplication with provincial and territorial roles and responsibilities. Provincial policy research funding can be limited and is focused on local rather than pan-Atlantic priorities. APRI funding is unique in its attention to pan-Atlantic economic development outcomes for programming. ACOA’s focus on Atlantic Canada allows it to respond to specific regional needs, opportunities and contextual factors while taking advantage of synergies between the provinces.

4.0 Findings: performance – effectiveness

The evaluation team assessed the effectiveness of the APRI program by examining: (1) evidence of contribution to the achievement of expected immediate outcomes in the PAC logic model, (2) timeliness of policy research, (3) value of facilitative activities that support project development, (4) approaches to reciprocal information sharing, and (5) use of social media and other technology.

Overall, the evaluation found that APRI contributes to the achievement of the immediate expected outcomes of PAC. Challenges impacting the success of the program are known to APRI management and it is apparent that many are being mitigated. Some factors would benefit from further investigation, particularly internal engagement, internal and external communication and knowledge mobilization.

4.1 Achievement of expected results

Evaluation Questions	Judgment Criteria	Key Findings
How well is APRI achieving its pan-Atlantic program objectives? Does APRI contribute to the immediate outcomes of PAC?	APRI is fulfilling the pan-Atlantic aspect of its mandate to undertake activities that have an Atlantic Canada scope (i.e. that impact more than one Atlantic province). There is sufficient qualitative and/or quantitative evidence to argue the contribution of APRI to the achievement of the immediate outcomes of PAC. The achievement of outcomes is similar to or greater than that observed in the previous evaluation (accounting for changes in context, processes and procedures).	#7: APRI investments are of a pan-Atlantic nature and contribute to the immediate outcomes of PAC, consistent with what was reported in the 2010 evaluation.

Evaluation Questions

Judgment Criteria

Key Findings

How well is APRI achieving its pan-Atlantic program objectives? Does APRI contribute to the immediate outcomes of PAC?

APRI is fulfilling the pan-Atlantic aspect of its mandate to undertake activities that have an Atlantic Canada scope (i.e. that impact more than one Atlantic province).

There is sufficient qualitative and/or quantitative evidence to argue the contribution of APRI to the achievement of the immediate outcomes of PAC.

The achievement of outcomes is similar to or greater than that observed in the previous evaluation (accounting for changes in context, processes and procedures).

#7: APRI investments are of a pan-Atlantic nature and contribute to the immediate outcomes of PAC, consistent with what was reported in the 2010 evaluation.

Finding #7: APRI investments are of a pan-Atlantic nature and contribute to the immediate outcomes of PAC, consistent with what was reported in the 2010 evaluation.

There was sufficient evidence to confirm that APRI contributes to the immediate expected outcomes reflected in the PAC logic model. In general terms, the program is expected to contribute to the PAC functions (i.e. policy, advocacy and coordination).

The 39 G&C and 26 O&M projects assessed during the evaluation reflected the pan-Atlantic mandate of APRI. In terms of linkages to the PAA, the majority of G&C projects focused on policy (89.7 per cent), with a smaller number of projects supporting advocacy (2.6 per cent) and coordination (7.7 per cent). The majority of O&M projects focused on policy (63 per cent), policy and coordination (16 per cent) or policy and advocacy (16 per cent). No O&M projects solely addressed advocacy or coordination. Key informants indicated that they believed that the intended outcomes of these projects were aligned with the PAC outcomes.

Given that the majority of G&C and O&M funding was allocated to policy-related efforts, and coordination to a lesser extent, the following section emphasizes exploration of these activities and outcomes.

Well-informed policy decisions

Some key informants indicated that it is hard to quantify the contribution of policy research to decision making as it is only one of many inputs into policy decisions. Further, policy research often provides knowledge that leads us to other questions that often need to be explored. Some internal key informants highlighted that opportunities for additional analysis of policy research exist within ACOA. They indicated that there needs to be more proactive planning that is oriented toward applying the knowledge gained from policy research and fostering the exploration of additional questions (see section 4.4).

All case studies highlighted the influence of APRI results on policies and programs. Although it is too early to determine the extent and impact of longer-term outcomes, there is the potential for further analyses by both ACOA and its partners.

The Functional Economic Regions project developed an evidence-based model to aid policy-makers and economic development practitioners in identifying what functional economic regions mean in an Atlantic Canadian context. The concept of functional economic regions provided a way to examine the linkages and flows that create interdependence among people, communities and regions. The outcomes of the project will equip policy-makers with tools to better support future local economic development in Atlantic Canada, and will assist practitioners in embracing the value of strategic partnerships for economic growth in the long term. The research has been shared with economic development partners at the federal, provincial and municipal levels as well as internationally.
The Bioenergy research project assessed the bioenergy assets of the region, taking into consideration new developments, technologies and opportunities that have emerged in recent times. The project built a business case for a bioenergy sector in the region and identified the economic impact of such activities. This project focused on an emerging sector, involved a regional expert advisory board, and contributed to the body of knowledge supporting the federal economic development agenda. Outputs have increased the body of knowledge on regional and economic development, and influenced and enhanced economic development policy and programming that impact the Atlantic region.
The Global Value Chains study improved knowledge of Atlantic Canada's trade flows from a value chain perspective, and provided information and advice on how Atlantic businesses can develop and manage their own value chains and/or become suppliers to global value chains. This project led to engagement with the business community and the development of a business-friendly brochure. In addition, the proponent disseminated the research broadly and made it public. ACOA also made use of the research and analysis, which helped inform and shape its international business development activities, including the Atlantic Gateway, and other enterprise development programs. The knowledge generated provided insight to the Agency on strategies and priorities to help Atlantic firms develop their own value chains and increase their sales by improving their participation in global value chains.
The SME Financing project was an O&M project through which ACOA contributed to a federal consortium of departments led by Industry Canada. It funded Statistics Canada to collect data on SME financing at the regional level. The intent was to survey these enterprises to determine the kind of debt, capital lease and equity financing SMEs rely on, as well as to gain information on attempts to obtain new financing. In addition, the survey gathered information on firm growth, engagement in international business activities, innovation and IP as well as owner characteristics. One immediate outcome of this project was that as a result of the analysis, a research gap was identified in the area of business succession. That is, the SME Financing data showed that Atlantic business owners are older than elsewhere nationally, implying that business succession planning is very important for Atlantic Canada.

Some external key informants highlighted that the “independence” of the proponent for APRI G&C projects supported the goal of producing credible and reliable knowledge for audiences other than ACOA (federal), including other public- and private-sector stakeholders. There are cases where policy research has led to linkages with future work by other stakeholders. For example, the knowledge gained from the Global Value Chains project completed in 2011 has subsequently been used by academics, governments and industry, which has further validated the usefulness and value of the research.

Atlantic interests are reflected in federal policies and programs (advocacy)

APRI projects have primarily focused on supporting the policy function within PAC, and internal key informants highlighted that the type of research conducted through APRI is less directly applicable to the advocacy function. Often, opportunities to provide input into emerging federal priorities through the advocacy function require quick response time. Conducting research such as APRI can take months or years to complete. However, knowledge from completed APRI studies can help the Agency’s advocacy unit to respond to emerging federal issues if the research topic falls within that subject area. According to APRI management, emerging federal priorities reflect the types of key priority areas that require proactive policy research and analysis.

Coordination of partners in addressing economic priorities

Many APRI projects involved collaboration with other Agency groups. For example, ACOA’s Atlantic Energy Office, which collaborates and coordinates with other federal and provincial departments, provided input on the biofuels (Bio-Atlantec) project.

Each of the case studies illustrated how APRI has supported ACOA’s policy coordination function by funding a network of external research partners and stakeholders with common interests focused on Atlantic Canadian economic development issues. By holding round tables and consulting with key stakeholders in Atlantic Canada, each case study project was able to build and maintain these critical relationships.

Examples of APRI project results that contribute to PAC activities include:

The Functional Economic Regions project: an important governance aspect of the research was the Federal-Provincial Advisory Committee established for this project. The purpose of the committee was to bridge research and practice. A key role of this committee was to support the researchers in grounding the research and as they disseminated the results to various economic development agencies and organizations in Atlantic Canada and more broadly;
The Bioenergy project: considerable knowledge and expertise was leveraged in the delivery of this project. In preparing the report, the project proponents engaged key stakeholders (e.g. producers, retailers, refineries, proponents, policy-makers and researchers) from all four provinces around the issue of bioenergy; and
The Global Value Chains project: two round tables were held with business representatives to solicit feedback on APEC’s preliminary research and to seek input on priorities to strengthen the region’s transportation and logistics system and on broader policies/programs to better support the development of global value chains in the region.

Case studies illustrated how APRI has contributed to PAC key outcomes through the Initiative’s support of long-term coordination and capacity-building with partners, in terms of:

the availability of knowledge (e.g. functional economic regions, bioenergy, global value chains, SME financing);
the development of state-of-the-art skills and the institutionalization of new analytical capacity (functional economic regions); and
the establishment of new stakeholder networks and relationships (e.g. functional economic regions, bioenergy, global value chains).

APRI also leverages policy/programs expertise from internal ACOA stakeholders to advance knowledge on key issues facing the region. Examples are: (a) energy initiatives such as the bioenergy project involved the Atlantic Energy Office and (b) the functional economic regions project involved Community Development program expertise at HO and the regional offices to inform the development of the community investment framework that was part of the Community Investment management action plan. These examples demonstrate positive steps in the pursuit of collaboration and leveraging expertise from other areas within the Agency; however, opportunities exist to further enhance internal engagement (see section 4.4).

4.2 Timeliness of policy research

Evaluation Questions	Judgment Criteria	Key Findings
In what ways has policy research been made available to decision-makers in a timely manner to act on economic issues?	Stakeholders indicate that decision-makers received policy research outputs in a timely manner to support decision making on economic issues.	#8: The nature of policy research often requires varying time frames for completion. There are opportunities to enhance proactive planning and timeliness of policy research.

Finding #8: The nature of policy research often requires varying time frames for completion. There are opportunities to enhance proactive planning and timeliness of policy research.

Key informants identified important contextual challenges with producing timely policy research. It takes time to conduct sound evidence-based research, particularly when developing the knowledge base required to initiate policy research and analysis. As APRI is proponent driven, proponents interested in undertaking and completing a study on a particular research topic do so within their own time frame, which can impact the timeliness of the research and product.

Key informants also related that if the findings of policy research are not timely or realistic, then the final output (often a report) loses some of its credibility and value. There are recent cases where timely research has been linked to ongoing decision making (e.g. SME financing) and future work (e.g. functional economic regions, global value chains, SME financing). There have also been cases where research was completed but, because of the timing of the completion of the policy research and analysis, did not optimize the benefits of the research to the desired extent. However, in cases where research was not yet completed, it was considered useful in contributing to a better informed body of knowledge and/or strengthening partnerships that supported the Agency’s decision-making process.

With APRI situated within the broader policy context, it is difficult to completely disentangle APRI planning from the broader PAC planning context and some evaluation questions addressed this broader perspective. Some internal key informants indicated that there are opportunities to enhance the timeliness and identification of overall policy research priorities, the tools to address them, and the optimal timelines for fostering utilization. Key informants also noted that it would be beneficial to have a more proactive approach to determining medium and long-term priorities.

In some cases, the successful and timely dissemination of policy research was linked to proactive knowledge mobilization plans and their implementation both during and after completion. In terms of APRI’s G&C-funded research, proponents own the IP/research and their contribution agreements with ACOA include communication and knowledge mobilization plans, which may include internal audiences (see section 4.4 for examples of dissemination, findings on knowledge mobilization and practical suggestions).

4.3 Value of facilitative activities

Evaluation Questions	Judgment Criteria	Key Findings
To what extent, and in what ways, do facilitative activities (e.g. engagement, outreach, relationship building, exploration, consensus building) support the achievement of broader program objectives such as the promotion of forward thinking around key, evolving and sometimes higher risk issues?	There is sufficient qualitative evidence (including examples) to argue that the APRI program has contributed to advancing forward thinking with respect to economic policy research capacity in Atlantic Canada.	#9: There is evidence that facilitative activities with key external partners and stakeholders for project development have supported the delivery of forward-thinking projects.

Finding #9: There is evidence that facilitative activities with key external partners and stakeholders for project development have supported the delivery of forward-thinking projects.

There was sufficient evidence to confirm that the APRI program, through facilitative activities led by program staff to develop projects, can act as a tool to promote collaboration, shared understanding of issues, and concerted action. This facilitative approach to project development and implementation helps ACOA build networks of partners and encourages dialogue and relationships that can be helpful going forward in other forums.

For example, internal key informants and case study project proponents indicated that there was a high level of engagement between the researchers and ACOA staff during project development. Each case study project represented an APRI investment that led to the promotion of forward thinking around key, evolving and sometimes higher risk issues.

Further, in terms of incrementality, key informants indicated that APRI funds external policy research studies in areas that are not otherwise supported (e.g. functional economic regions, bioenergy, tidal energy), and it allows proponents to study topics that otherwise could not be studied. As mentioned previously, the three G&C case study projects would likely not have proceeded without the funding contribution from the APRI program and the fourth, funded through O&M, would not have incorporated an Atlantic lens without ACOA support (e.g. the ACOA contribution to a survey of SMEs allowed for oversampling within the four Atlantic provinces to ensure there was a representative sample to support data analyses specific to Atlantic Canada).

4.4 Approaches to reciprocal information sharing

Evaluation Questions	Judgment Criteria	Key Findings
To what extent, and in what ways, has there been two-way communication (i.e. reciprocal information sharing) between the APRI program and both internal and external stakeholders, including HO and regional program staff, to support awareness of APRI and stakeholder objectives, engagement in terms of priority setting, and dissemination and use of outputs, including any operational implications?	There is sufficient qualitative and/or quantitate evidence to argue that internal and external communication activities have been effective. The communications activities, outputs and outcomes have been enhanced since the previous evaluation (accounting for changes in context, processes and procedures).	#10: Enhancing stakeholder awareness and engagement is a component of the APRI program. APRI program activities have led to greater awareness of its objectives and outputs with both internal and external stakeholders. #11: There are opportunities to further enhance internal engagement on priorities and outputs. Internal and external communication could be enhanced, in particular, to leverage opportunities to disseminate information about APRI program outputs.

Evaluation Questions

Judgment Criteria

Key Findings

To what extent, and in what ways, has there been two-way communication (i.e. reciprocal information sharing) between the APRI program and both internal and external stakeholders, including HO and regional program staff, to support awareness of APRI and stakeholder objectives, engagement in terms of priority setting, and dissemination and use of outputs, including any operational implications?

There is sufficient qualitative and/or quantitate evidence to argue that internal and external communication activities have been effective.

The communications activities, outputs and outcomes have been enhanced since the previous evaluation (accounting for changes in context, processes and procedures).

#10: Enhancing stakeholder awareness and engagement is a component of the APRI program. APRI program activities have led to greater awareness of its objectives and outputs with both internal and external stakeholders.

#11: There are opportunities to further enhance internal engagement on priorities and outputs. Internal and external communication could be enhanced, in particular, to leverage opportunities to disseminate information about APRI program outputs.

Finding #10: Enhancing stakeholder awareness and engagement is a component of the APRI program. APRI program activities have led to greater awareness of its objectives and outputs with both internal and external stakeholders.

All key informants agreed that ongoing engagement between APRI staff and both internal and external stakeholders leads to better findings (e.g. knowledge of key issues) and ownership of the results of APRI projects. There is sufficient evidence to confirm that since the last evaluation, there has been enhanced reciprocal information sharing between APRI staff and internal and external stakeholders. This two-way communication has supported better awareness of the APRI program, engagement to support priority setting and broader dissemination of outputs.

Internal engagement

Key informants indicated that there is two-way dialogue between internal policy and program functions. This ongoing dialogue results in the sharing or exchange of information relevant to both parties and can take place at any point in the project life cycle.

In response to the evaluations of APRI (2010) and PAC (2012), the PRWG was established as a forum to discuss research across the Agency. APRI proposals are shared with the PRWG, which helps to build stronger and more robust research projects by exposing the research proposal to the Agency’s policy research capacity (see section 5.2.2), which is distributed across the Agency and differs from region to region. In addition, the regional requirements for policy research are often focused at a provincial level whereas APRI’s mandate requires a pan-Atlantic lens. Regional participants bring a regional lens to pan-Atlantic project discussions and also share information about their regional projects.

The APRI team, through the Director of Strategic Policy and the DG Policy, engages with the DG PAC and other senior management committees (e.g. the ExCom and the DG, Operations) on relevant research. Similarly, the Director and DG engage the APRI team to respond to issues of priority to ACOA management, including recent topics such as the Canada-European Union Trade Agreement and business succession.

Internal dissemination

Internal key informants indicated that all major APRI projects have included some dissemination within ACOA, including at the working level, with senior management and with the minister. It was noted by some key informants that working-level dissemination has sometimes been targeted more to staff working in the policy function rather than to all potentially interested parties within the organization, or more broadly to all staff.

A number of activities have supported the internal dissemination of results:

APRI project reports are available through ACOA’s library services.
Proponents of G&C projects often post their studies on their own external websites, which can be accessed by ACOA staff.
Since it was established in 2013, the PRWG has been an effective venue in which to share information about completed projects. A SharePoint site that serves as a repository for policy research activities from across the organization (including project name, status, short description, cost and proponent) was launched and is maintained in part through input from the PRWG.
Recently, for one major project (functional economic regions), a series of presentations and webinars were organized inviting broad internal audiences to attend so that project results could be widely communicated by the proponents. The method of communication and the inclusion of all staff were well received and were viewed as a best practice.
APRI staff have indicated that the communications plan for major projects recently has evolved to include not only briefing notes to senior officials but a one-pager with key findings that can be shared with others, including working level, external partners and senior management.

External engagement

The approach to external engagement has been strategic. Over the past two years, the APRI team has held a number of targeted outreach sessions with potential external researchers from three provinces across the region to build awareness of the program within the research community. Universities visited include the University of New Brunswick, the University of Prince Edward Island, Mount Allison University and Memorial University of Newfoundland.

There is evidence that APRI management supports strategic external engagement. However, some key informants believe that there are challenges in certain regions where there are fewer opportunities for communicating APRI’s function and outputs. Some key informants indicated that it would be beneficial for the APRI team, in conjunction with the PRWG, to continue their strategic engagement efforts.

Finding #11: There are opportunities to further enhance internal engagement on priorities and outputs. Internal and external communication could be enhanced, in particular, to leverage opportunities to disseminate information about APRI program outputs.

Internal engagement

Some key informants indicated that there are organizational challenges to, as well as opportunities for, enhanced internal engagement in the planning and delivery of research projects.

Key informants raised the issue of inconsistent awareness of the policy research planning among interested parties across the Agency. Staff turnover has been a challenge – sometimes leading to a need for repeated engagement across some areas. There is concern by some key informants that gaps in engagement could lead to overlap or even duplication of research efforts across regions. It was suggested that this concern could be mitigated by involving program staff and other interested parties at the beginning of the development process of policy research projects, in addition to continuing to work with them to apply what was learned upon completion of the study. More direct engagement through the PRWG was another suggestion made by key informants.

Although there is evidence to show that initiatives have been undertaken to share information, it was noted by key informants that quick and easy access to information about ACOA's policy research studies continues to be a challenge. Dissemination efforts reach the Agency’s policy functions but do not always reach programs and other interested parties. Some key informants indicated that the current SharePoint site is not easily accessible and suggested that better integration with the ACOA library might be considered.

The document review and key informant interviews suggest that the APRI team and the ACOA library are exploring new technology available to them to enhance internal engagement on the APRI program and outputs (e.g. SharePoint) and efforts continue to optimize and build awareness on how to use the various features of new tools.

In 2011, an APRI Internal Communications Strategy was developed in partnership with the Communication Branch. A number of activities have been undertaken, including: development of a repository of APRI projects managed within Library Services and development of a dissemination plan for each project. Some proposed actions in the communications plan were deemed not to be viable over time, including publishing the APRI program description and featuring APRI outputs on ACOA’s website. Some proposed actions have yet to be undertaken, including the posting of articles on APRI projects on ACOA’s Rendezvous (intranet) site.

Internal and external dissemination and knowledge mobilization

Knowledge mobilization is defined as getting the right information to the right people, in the right format at the right time, so as to influence decision making. Knowledge mobilization includes dissemination, knowledge transfer and knowledge translation.[v] A carefully planned and executed knowledge translation and communication strategy is vital to the work of a policy research unit. Without it, good policy research can go unnoticed or, worse, unused[vi]

Key informants indicated that the third-party dissemination of results (particularly by proponents for G&C investments) is an effective way to share knowledge with key groups of stakeholders. There have been some examples of both broad and targeted dissemination by the proponent that have proven to be successful and can be considered best practices:

Most case study projects (e.g. functional economic regions, bioenergy, global value chains) included internal and external face-to-face dissemination by the proponent, either through presentations or round tables (see section 4.1).
Some case studies showcased the use of products tailored to specific target audiences. For example, the global value chains project developed an industry brochure to target dissemination to that particular audience.
All case study projects included the posting of products on the Web.

In addition, there was evidence of promising dissemination practices under way:

pursuing opportunities for face-to-face interactions with a subject matter expert to discuss the findings, their application and policy implications;
seeking ACOA program suggestions and support for additional knowledge mobilization on APRI deliverables to external stakeholders; and
building into contribution agreements requirements for user-friendly, targeted publications, dissemination events tailored to audiences (i.e. face to face if possible), and provisions for proactive dissemination to relevant stakeholders.

A number of key informants suggested implementing or continuing existing practices to support enhanced knowledge mobilization:

providing short, digestible summaries of key research projects, results and implications;
leveraging the knowledge, skills, expertise, and networks of other PAC policy analysts to support knowledge mobilization of APRI-funded projects;
holding regular face to face and/or virtual engagement at various levels (e.g. ACOA senior management and program staff, other orders of government, other stakeholders) to enhance timeliness of the identification of research priorities as well as to disseminate findings; and

These suggestions are consistent with best practices noted in the document and literature, which emphasize early sharing of interim research findings with policy-makers (i.e. finding opportunities for information exchange beyond the product stage)[vii] for optimal application of research finding.[viii]

4.5 Use of social media and other technology

Evaluation Questions	Judgment Criteria	Key Findings
To what extent, and how, can social media and other technology be used to engage with stakeholders and support communication and dissemination of policy research findings and conclusions?	Social media and other technology has been used to support the effective communication and dissemination of policy research findings and conclusions.	#12: Social media has been used to some extent to engage stakeholders and disseminate policy research outputs.

Finding #12: Social media has been used to some extent to engage stakeholders and disseminate policy research outputs.

ACOA has begun to employ the use of social media for information dissemination. Examples include the use of Twitter to announce some major financial investments and the use of the Rendezvous site to share knowledge internally with ACOA staff. Within the Government of Canada generally, the use of social media is still a relatively new approach to sharing information and opportunities to adopt it are still evolving.

Recognizing that the use of social media is a relatively new phenomenon and that these projects were initiated before 2013-2014, the four case studies reviewed showed limited evidence of the use of social media to disseminate results from APRI-funded projects. The one exception was the use of Twitter at a conference where results of the global value chain project were disseminated. However, external key informants (proponents) see some value in using social media to disseminate information about the results of APRI-funded projects in the future.

It was also noted that, for all G&C projects, the APRI program has highlighted that the proponents hold the dissemination responsibilities. ACOA is not the owner of the resultant IP, therefore contribution agreements reflect the communication requirements of the Government of Canada. There is no current approach to guide the level of information that can and should be published in this format for policy research projects funded through APRI.

5.0 Findings: performance – efficiency and economy

Efficiency and economy focus on the optimization of resources and activities. Optimization is facilitated by the existence of strong management structures, practices and mechanisms, including the existence and effectiveness of governance structures, consideration of alternative delivery mechanisms, lessons learned, best practices and performance measurement data. Overall, the evaluation found that APRI demonstrates efficiency and economy in the utilization of resources[ix] and in the management of activities in support of policy research in Atlantic Canada.

One of the tools used by ACOA to deliver external policy research activity, APRI is an efficient and economical program. There are opportunities to further strengthen current operational efficiencies, including the enhancement of APRI’s operational processes and tools, and continued and enhanced implementation of the best practices identified in this evaluation.

As the APRI logic model and the PMS had not been updated since the 2010 evaluation, limited outcome level performance measurement data were collected. It is anticipated that the upcoming PAC logic model exercise will support APRI in better planning and monitoring the program’s results and indicators.

The evaluation assessed efficiency and economy by examining evidence of: (1) efficient utilisation of resources in terms of planning, governance and delivery; (2) consideration of alternative delivery mechanisms, lessons learned and best practices; and (3) the availability and use of performance measurement information.

5.1 Efficient utilization of resources

Evaluation Question	Judgment Criteria	Key Findings
How effective are the mechanisms and structures within ACOA that facilitate policy research governance (e.g. planning, conducting, dissemination and use)? To what extent is the Agency making optimal use of the knowledge assets generated through policy research? What are the barriers to this optimization?	Qualitative evidence that ACOA has structure/mechanisms in place to ensure that the most efficient and economical means are being used to administer the programming. Governance processes are appropriate: there is a clear delineation of roles and responsibilities; and planning and decision-making processes are clear and streamlined. There is qualitative evidence that the knowledge assets generated through policy research are being optimized (e.g. shared, accessible, etc.). There is evidence that factors (internal and/or external) that are facilitating or impeding the success of APRI are known, and mitigation strategies are implemented where appropriate.	#13: Mechanisms are in place that support the efficient and economical governance and delivery of APRI.

Evaluation Question

Judgment Criteria

Key Findings

How effective are the mechanisms and structures within ACOA that facilitate policy research governance (e.g. planning, conducting, dissemination and use)? To what extent is the Agency making optimal use of the knowledge assets generated through policy research? What are the barriers to this optimization?

Qualitative evidence that ACOA has structure/mechanisms in place to ensure that the most efficient and economical means are being used to administer the programming.

Governance processes are appropriate: there is a clear delineation of roles and responsibilities; and planning and decision-making processes are clear and streamlined.

There is qualitative evidence that the knowledge assets generated through policy research are being optimized (e.g. shared, accessible, etc.).

There is evidence that factors (internal and/or external) that are facilitating or impeding the success of APRI are known, and mitigation strategies are implemented where appropriate.

#13: Mechanisms are in place that support the efficient and economical governance and delivery of APRI.

Finding #13: Mechanisms are in place that support the efficient and economical governance and delivery of APRI.

5.1.1 Structures and mechanisms that support efficiency and economy

Multiple mechanisms currently exist to support efficient and economical program delivery and have been enhanced by the contributions of the PRWG; by dialogue with users of policy research and information sharing that supports effective policy research activities; and through collaborative relationships between key funding partners and ACOA.

Governance and collaboration mechanisms

Clear governance promotes improved efficiency and economy through well-defined roles, decision making and procedures. According to the Institute on Governance, “governance determines who has power, who makes decisions, how other players make their voices heard and how account is rendered.”[x]

Supporting the Agency’s governance structure is the DG PAC. The purpose of the committee is to ensure that ACOA senior management receives strategic and timely advice on, and identification of, policy issues facing the region and the Agency. The committee oversees the Agency’s PAC activities using an integrated team approach to share information, initiate and guide policy activities, and engage Agency resources in all regions as well as at HO on corporate policy priorities and strategic planning.

ACOA has taken several steps to enhance the efficiency of the governance and delivery structure of its policy research activities, including the establishment of the PRWG in June 2013. The PRWG has a direct reporting relationship to the DG PAC and has a mandate to support the committee in its deliberations on key policy issues impacting the Atlantic economy and to ensure that the body of knowledge to support effective strategic decision making is available. The creation of this committee is a deliberate approach to advancing collaboration and is seen by key informants as critical to strengthening the governance and delivery of policy research, including the external policy research conducted through APRI.

In addition, recent cross-cutting discussions between DG PAC committee members and the Agency’s Directors General of Operations have created another element to support effective governance in terms of collaboration and sharing of regional and program perspectives. This is seen as a good practice by key informants and one that will enhance overall policy and program communication.

Ensuring collaboration through stakeholder engagement (both internal and external) across all phases of the project (development, implementation and dissemination) was a key factor in the success of the four case studies that were compiled.

Planning and budgeting mechanisms

Coordinated planning and budgeting mechanisms support the efficiency and economy of APRI programming. The APRI team has established a due diligence regime that recognizes and subsequently helps develop the strongest project proposals, which are then vetted through the new PRWG, consisting of ACOA’s senior policy members and internal policy experts. Key informants see this as an efficient step in the planning process since it allows for regional input prior to the project approval.

In terms of budget oversight, APRI received $3.4 million during the five-year period of the evaluation, which allows the program to deliver on its policy research objectives (see section 2.4.1 for funding breakdown). According to the Agency’s QAccess database, ACOA supported an annual average of eight G&C projects during the period of this evaluation, with an overall annual average of $560,000 expended. The average project size for G&C projects was $71,699. As with other Transfer Payment Programs, APRI complies with the program terms and conditions as required by the TB as well as the internal controls established by ACOA.

The APRI team manages annual average budgets of $86,964 for O&M projects and approximately $184,000 for the salaries of 2.1 full-time equivalents. The number and value of O&M projects vary by year. The O&M data provided during the period of this evaluation indicate an annual average of five projects funded with an overall annual average of $87,000 expended. The average project size for O&M projects was $16,724.

Delivery mechanisms

The delivery of funding through APRI is an efficient and economical programming mechanism. APRI is a unique, pan-Atlantic program that is able to support external policy research projects that may not otherwise be funded. Additional APRI strengths include the small dedicated and experienced team that manages the program and the critical expertise provided by ACOA’s regional offices and members of the PRWG.

There is evidence that certain strategies have been enacted to improve the delivery of APRI since the previous evaluation, including:

recent dialogue by the PRWG intended to encourage greater awareness, collaboration and dialogue around regional priorities;
focused outreach to internal and external stakeholders by the APRI team. The purpose of the outreach is to develop strategic policy research opportunities and partners through enhanced engagement; and
improved dissemination of policy research results with the use of WebEx presentations and targeted publications on key research pieces, as identified by two case studies: functional economic regions and global value chains. However, there remains a need to formalize the dissemination approach and take it one step further to the optimization of knowledge mobilization – use of the product – the “so what” element of the research.

Further opportunities exist to address ongoing challenges faced by APRI. For example, the previous evaluation stated that APRI was not being promoted extensively internally or externally. In 2014, regional key informants indicated that there remains a lack of awareness of the APRI program, and its potential to support regional research priorities, and a lack of dissemination and use of the end products.

Dissemination and knowledge mobilization

ACOA ensures that a dissemination and communication strategy is incorporated into the contribution agreements for all major research projects. Third-party dissemination of results, particularly by proponents of G&C investments, is an effective way to share knowledge with broad and key groups of stakeholders. Case study informants confirmed the value of knowledge mobilization in terms of strategically translating and packaging the knowledge so that it is easily accessible and able to be applied by stakeholders (see section 4.4).

There is evidence that recent progress has been made in disseminating research results (e.g. WebEx), and there is recognition of the importance of sharing key research pieces with both internal and external stakeholders in the future. However, ACOA does not have a specific approach for knowledge mobilization. APRI could benefit from best practices in knowledge mobilization as well as the expertise of ACOA’s regional policy and programs staff and the members of the PRWG.

5.2 Alternative modes of delivery, lessons learned and best practices

Evaluation Questions	Judgment Criteria	Key Findings
Is there a more cost-effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned? What can APRI learn from other economic policy funding mechanisms in other regional development agencies or similar organizations?	Delivery costs compare favourably to the delivery costs as calculated during the previous evaluation and to the delivery costs of similar programming offered in other jurisdictions. Lessons learned and best practices are identified regularly. There is evidence that program management has considered and continues to explore alternative modes of delivery.	#14 APRI is considered a cost-effective mechanism in the support of economic development in Atlantic Canada. There are opportunities to enhance the efficiencies of operational processes and tools.

Evaluation Questions

Judgment Criteria

Key Findings

Is there a more cost-effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned? What can APRI learn from other economic policy funding mechanisms in other regional development agencies or similar organizations?

Delivery costs compare favourably to the delivery costs as calculated during the previous evaluation and to the delivery costs of similar programming offered in other jurisdictions.

Lessons learned and best practices are identified regularly.

There is evidence that program management has considered and continues to explore alternative modes of delivery.

#14 APRI is considered a cost-effective mechanism in the support of economic development in Atlantic Canada. There are opportunities to enhance the efficiencies of operational processes and tools.

Finding #14: APRI is considered a cost-effective mechanism in the support of economic development in Atlantic Canada.There are opportunities to enhance the efficiencies of operational processes and tools.

5.2.1 Alternative modes of delivery

Evaluation results do not identify a more effective or efficient approach to the delivery of APRI, as almost all key informants agreed that using APRI to provide policy research funding was both an appropriate and a cost-effective way to support economic development in Atlantic Canada. Many interviewees stated that the strength of the program resides in APRI’s ability to support policy research that may not otherwise be funded, the efficiency of the small APRI team, the PRWG, ACOA’s internal policy and program expertise, ongoing engagement and the availability of both G&C and O&M funding from which to draw.

As mentioned in the relevance section of this report, there are other federal RDAs with similar policy research mandates. It is difficult to compare these other agencies to ACOA, and the work done by APRI, since they either do not have a similar program or their structures are significantly different.

There is evidence that APRI management explores alternative modes of delivery through the efforts of the PRWG and their engagement with internal and external stakeholders.

Delivery costs

According to key informants, APRI is considered a cost-effective policy research development model. Based on the Agency’s financial data, resources allotted to APRI are similar to those noted in previous evaluations.

Leveraging

All key informants discussed the funding challenges that currently exist in the policy research community, and this point was further proven by the APRI data review, which confirmed that ACOA is often the only financial contributor apart from the proponents. Three of the case study clients stated that their projects would not have proceeded without ACOA assistance and the fourth noted that without APRI support, research at the Atlantic level would not have been possible. APRI management works with the proponents to attract other potential funding partners. According to the TB Policy on Stacking, proponents of APRI funding are required to contribute a minimum of 10 per cent to the projects and this is normally in the form of in-kind contributions.

Between 2009-2010 and 2013-2014, ACOA supported 39 G&C projects, contributing $2.8 million, with clients leveraging $1.3 million, normally from within their own organizations. For every $1.00 invested by the Agency in support of policy research, an additional $0.46 was contributed by other sources/proponents.

Best practices generated by APRI management

There is evidence that the APRI team documents best practices. For example, its working relationship with the Atlantic Energy Office is seen by the APRI team as a best practice in terms of the level of engagement and collaboration from both sides to strengthen policy research in Atlantic Canada. A list of best practices was compiled by APRI management and provided for the purposes of analysis to augment the information gathered for this evaluation.

5.2.2 Best practices generated from the evaluation

The evaluation identified the following best practices supporting efficiencies, economy and effectiveness of APRI programming:

methods of dissemination on the functional economic regions and global value chains projects;
external promotion of APRI through outreach, engagement and knowledge mobilization;
consultation efforts of the PRWG to support Agency-wide collaboration; and
continuing to invite representatives, on an ad hoc basis, from program management to contribute to the PRWG to allow for greater communication.

5.3 Adequacy of performance measurement

Evaluation Questions	Judgment Criteria	Key Findings
Is there appropriate performance measurement in place? If not, what steps are required to support the development and implementation of more effective performance measurement activities? If yes, is performance measurement information being used to inform senior management decisions?	Performance data is being collected and is available for use during the evaluation. Qualitative evidence that performance measurement data is adequate and is effective in reporting on the achievement of outcomes. Qualitative evidence that performance measurement data is considered useful and is being used in decision making.	#15: Efforts are under way to revise and update the logic model and PMS. There has been limited outcome level performance measurement data collected.

Evaluation Questions

Judgment Criteria

Key Findings

Is there appropriate performance measurement in place? If not, what steps are required to support the development and implementation of more effective performance measurement activities? If yes, is performance measurement information being used to inform senior management decisions?

Performance data is being collected and is available for use during the evaluation.

Qualitative evidence that performance measurement data is adequate and is effective in reporting on the achievement of outcomes.

Qualitative evidence that performance measurement data is considered useful and is being used in decision making.

#15: Efforts are under way to revise and update the logic model and PMS. There has been limited outcome level performance measurement data collected.

Finding #15: Efforts are under way to revise and update the logic model and the PMS. There has been limited outcome level performance measurement data collected.

The evaluation examined the collection, adequacy and effectiveness of APRI performance measurement data and found limitations in the mechanisms used to capture the data and in the level of outcome data available.

Progress has been made since the 2010 evaluation in terms of a SharePoint repository where information about policy research projects can be captured. For the purpose of this evaluation, the repository included one project funded in 2012-2013, with the remainder funded in 2013-2014. Results are not yet being captured on this site. In addition, performance measurement information collected to date consists largely of output data (e.g. QAccess and GX data on funded projects). Based on the various measurement tools examined, it appears that minimal data is currently systematically collected on APRI program outcomes.

The APRI program logic model outlines program objectives, activities, outputs and outcomes. The logic model has not been updated since the last evaluation in 2010. At that time, suggestions were made to streamline some of the stated outcomes. To address this need, key informants indicated that APRI will be considered within the broader PAC logic model and performance measurement exercise taking place in 2015. It should be noted that although the evaluation has identified limitations in the level and quality of performance measurement data collected for APRI, the evaluation team recognizes that APRI is a small program with limited resources. Therefore, consideration should be given to calibrating performance measurement efforts to account for program materiality, risk and the information needs of management.

Opportunities for improvements to performance measurement include:

an updated logic model with performance indicators appropriately defined for APRI and linked to PAC outcomes, with a focus on the immediate outcome level;
an approach to identify and monitor program outcomes, including knowledge mobilization; and
a systematic and ongoing electronic tracking of O&M investments.

6.0 Conclusions and recommendations

The APRI evaluation conclusions and recommendations were identified from analysis of the key findings presented in this report. They were validated through discussion and consultation with the evaluation working group and the PRWG. Table 7 shows the links between key findings, conclusions and recommendations.

6.1 Conclusions

The following three conclusions were identified by the evaluation team based on the evidence presented in this report:

The APRI program is relevant. There is an ongoing need for policy research funding to address critical gaps in knowledge about the future of economic development in Atlantic Canada. Engagement and research activities supported by the program are aligned with the mandate and priorities of the Government of Canada and the Agency. APRI’s unique nature and ability to support broad policy research activities allows the program to complement rather than duplicate other ACOA/regional programs.
ACOA plays a key role in supporting the development of economic policy research and networking opportunities in the Atlantic region. APRI activities have been successful in supporting the immediate outcomes of the PAC branch.
- Steps have been taken to improve communication and engagement since the 2010 APRI evaluation, including the establishment of the PRWG and the development of research priorities. Opportunities exist to further enhance internal engagement on priorities and outputs and to improve internal and external communication. In particular, additional efforts to reach out to program areas and other interested parties to establish or validate research priorities, and to share information on completed policy research activities across ACOA, would enhance the effectiveness of this engagement.
- Knowledge mobilization is a critical component in transferring information to decision-makers in an increasingly complex policy environment. There are opportunities to enhance the knowledge mobilization of outputs from APRI-funded projects with both internal and external audiences.
ACOA recognizes the importance of efficient planning and governance in the delivery of APRI and has made progress in terms of establishing the PRWG and encouraging collaborative relationships between key funding partners and ACOA.
- As one of the tools used by ACOA to deliver policy research activities, APRI is an efficient and economical program. There are opportunities to further strengthen current operational efficiencies, including the enhancement of APRI’s operational processes and tools, and continued and enhanced implementation of the best practices identified in this evaluation.
- As the APRI logic model and the PMS had not been updated since the 2010 evaluation, limited outcome level performance measurement data were collected. It is anticipated that the upcoming PAC logic model exercise will support APRI in better planning and monitoring the program’s key results and indicators.

6.2 Recommendations

In the spirit of continuous improvement, it is recommended that APRI management:

Build on recent strategic engagement efforts by seeking strategic input from ACOA Programs and other interested parties in establishing policy research priorities and sharing information about past and current policy research.(Internal Engagement)
Explore opportunities to further enhance the knowledge mobilization of outputs from APRI-funded projects with both internal and external audiences.(Knowledge Mobilization)
In tandem with current PAC efforts, calibrate the performance measurement approach to better inform the APRI program and Agency decision making by identifying, tracking and analyzing key indicators that focus on monitoring both program outputs and immediate outcomes. (Performance Measurement)

Table 7: Alignment of APRI Evaluation Findings, Conclusions and Recommendations

SUMMARY OF KEY FINDINGS	SUMMARY OF CONCLUSIONS	RECOMMENDATIONS
RELEVANCE	The APRI program is relevant. There is an ongoing need for policy research funding to address critical gaps in knowledge about the future of economic development in Atlantic Canada. Engagement and research activities supported by the program are aligned with the mandate and priorities of the Government of Canada and the Agency. APRI’s unique nature and ability to support broad policy research activities allows the program to complement rather than duplicate other ACOA/regional programs. ACOA plays a key role in supporting the development of economic policy research and networking opportunities in the Atlantic region. APRI activities have been successful in supporting the immediate outcomes of the PAC programming. Steps have been taken to improve communication and engagement since the 2010 APRI evaluation, including the establishment of the PRWG and the development of research priorities. Opportunities exist to further enhance internal engagement on priorities and outputs and to improve internal and external communication. In particular, additional efforts to reach out to program areas and other interested parties to establish or validate research priorities, and to share information on completed policy research activities across ACOA, would enhance the effectiveness of this engagement. Knowledge mobilization is a critical component in transferring information to decision-makers in an increasingly complex policy environment. There are opportunities to enhance the knowledge mobilization of outputs from APRI-funded projects with both internal and external audiences. ACOA recognizes the importance of efficient planning and governance in the delivery of APRI and has made progress in terms of establishing the PRWG and encouraging collaborative relationships between key funding partners and ACOA. As one of the tools used by ACOA to deliver external policy research activity, APRI is an efficient and economical program. There are opportunities to further strengthen operational efficiencies, including the enhancement of APRI’s operational processes and tools, and continued and enhanced implementation of the best practices identified in this evaluation. As the APRI logic model and the PMS had not been updated since the 2010 evaluation, limited outcome level performance measurement data were collected. It is anticipated that the upcoming PAC logic model exercise will support APRI in better planning and monitoring the program’s results and indicators.	Recommendation 1: Internal Engagement: Build on recent strategic engagement efforts by seeking strategic input from ACOA Programs and other interested parties in establishing policy research priorities and sharing information about past and current policy research. Recommendation 2: Knowledge Mobilization: Explore opportunities to further enhance the knowledge mobilization of outputs from APRI-funded projects with both internal and external audiences. Recommendation 3: Performance Measurement: In tandem with current PAC efforts, calibrate the performance measurement approach to better inform the APRI program and Agency decision making by identifying, tracking and analyzing key indicators that focus on monitoring both program outputs and immediate outcomes.
There continues to be a need for policy research to support stronger economic growth in Atlantic Canada.
There are limited opportunities for researchers to access economic development policy research funding in Atlantic Canada.
APRI programming is responsive to existing and emerging policy research needs.
Economic development in Atlantic Canada, and policy research to support it, is a priority for the Government of Canada and ACOA.
There is a leadership role for the Government of Canada and ACOA in policy research related to economic development in Atlantic Canada.
APRI activities complement rather than duplicate other ACOA programming as well as other economic policy research programming in the region.
EFFECTIVENESS
APRI investments are of a pan-Atlantic nature and contribute to the immediate outcomes of PAC, consistent with what was reported in the 2010 evaluation.
The nature of policy research often requires varying time frames for completion. There are opportunities to enhance proactive planning and timeliness of policy research.
There is evidence that facilitative activities with key external partners and stakeholders for project development have supported the delivery of forward-thinking projects.
Enhancing stakeholder awareness and engagement is a component of the APRI program. APRI program activities have led to greater awareness of its objectives and outputs with both internal and external stakeholders.
There are opportunities to further enhance internal engagement on priorities and outputs. Internal and external communication could be enhanced, in particular, to leverage opportunities to disseminate information about APRI program outputs.
Social media has been used to some extent to engage stakeholders and disseminate policy research outputs.
EFFICIENCY AND ECONOMY
Mechanisms are in place that support the efficient and economical governance and delivery of APRI.
APRI is considered a cost-effective mechanism in the support of economic development in Atlantic Canada. There are opportunities to enhance the efficiencies of operational processes and tools.
Efforts are under way to revise and update the logic model and the PMS. There has been limited outcome level performance measurement data collected.

Appendix A – APRI evaluation management action plan

Recommendations	Management Responses	Planned Actions	Responsibility	Target Date
1. Build on recent strategic engagement efforts by seeking strategic input from ACOA Programs and other interested parties in establishing policy research priorities and sharing information about past and current policy research. (Internal Engagement)	Agree. Engaging with internal and external policy and program stakeholders continues to be a priority for APRI. APRI management will continue to build upon its recent engagement efforts with Agency Policy and Program officials and with the external policy research community on strategic policy research needs and issues facing Atlantic Canada.	APRI Management will expand its engagement efforts with internal and external stakeholders. Planned activities include: continue to hold strategic engagement and outreach sessions externally with the policy research community across Atlantic Canada to raise awareness of the APRI program and seek their input on the identification of policy research priorities; leverage the PRWG to continue to expand the two-way dialogue between policy and ACOA programs on policy research priorities; and engage with policy and program officials in the Agency to develop policy research priorities that could be used in part to guide decisions on APRI project funding.	APRI Management supported by the DG PAC and the PRWG.	April 2016 Ongoing Ongoing
2. Explore opportunities to further enhance the knowledge mobilization of outputs from APRI-funded projects with both internal and external audiences. (Knowledge Mobilization)	Agree. Knowledge mobilization and dissemination of results to the appropriate audiences is important to APRI. APRI will continue to support both internal Agency-wide dissemination while also supporting proponents in implementing their communications plans.	APRI management will pro-actively explore opportunities to enhance the dissemination of APRI program outputs. In addition to the program’s current knowledge mobilization efforts, APRI management will: work with research project proponents to ensure projects continue to include a pro-active knowledge mobilization plan that incorporates social media, where possible; build on recent best practices to support internal dissemination of results, such as the development of presentations, webinars and informational one-page summaries of research projects; develop and disseminate a regular APRI update to internal audiences and targeted external partners to share information on APRI policy research findings and conclusions; and provide training and support to help Agency personnel better navigate the Agency’s Library, which includes APRI research studies.	APRI Management supported by ACOA Library, the PRWG and the DG PAC.	April 2016 Ongoing
3. In tandem with current PAC efforts, calibrate the performance measurement approach to better inform the APRI program and Agency decision making by identifying, tracking and analyzing key indicators that focus on monitoring both program outputs and immediate outcomes. (Performance Measurement)	Agree. An up-to-date, calibrated PMS that integrates APRI PM requirements within the overall PAC strategy will help to strengthen PM practices for APRI and for PAC.	DG PAC is developing a revised PMS for PAC that incorporates the APRI. Given APRI’s low risk and materiality, this new strategy will be calibrated in such a way that the key performance indicators and expected results associated with APRI’s key activities (i.e. research studies, engagement and networking events) will support results-based management practices especially as they relate to program outputs and immediate outcomes. APRI Management will also work with proponents so they are aware of the expected results and key indicators so that project reporting mechanisms are in place to support this new strategy.	DG PAC supported by APRI team.	April 2016

Recommendations

Management Responses

Planned Actions

Responsibility

Target Date

1. Build on recent strategic engagement efforts by seeking strategic input from ACOA Programs and other interested parties in establishing policy research priorities and sharing information about past and current policy research. (Internal Engagement)

Agree. Engaging with internal and external policy and program stakeholders continues to be a priority for APRI. APRI management will continue to build upon its recent engagement efforts with Agency Policy and Program officials and with the external policy research community on strategic policy research needs and issues facing Atlantic Canada.

APRI Management will expand its engagement efforts with internal and external stakeholders. Planned activities include:

continue to hold strategic engagement and outreach sessions externally with the policy research community across Atlantic Canada to raise awareness of the APRI program and seek their input on the identification of policy research priorities;
leverage the PRWG to continue to expand the two-way dialogue between policy and ACOA programs on policy research priorities; and
engage with policy and program officials in the Agency to develop policy research priorities that could be used in part to guide decisions on APRI project funding.

APRI Management supported by the DG PAC and the PRWG.

April 2016

Ongoing

2. Explore opportunities to further enhance the knowledge mobilization of outputs from APRI-funded projects with both internal and external audiences. (Knowledge Mobilization)

Agree. Knowledge mobilization and dissemination of results to the appropriate audiences is important to APRI. APRI will continue to support both internal Agency-wide dissemination while also supporting proponents in implementing their communications plans.

APRI management will pro-actively explore opportunities to enhance the dissemination of APRI program outputs. In addition to the program’s current knowledge mobilization efforts, APRI management will:

work with research project proponents to ensure projects continue to include a pro-active knowledge mobilization plan that incorporates social media, where possible;
build on recent best practices to support internal dissemination of results, such as the development of presentations, webinars and informational one-page summaries of research projects;
develop and disseminate a regular APRI update to internal audiences and targeted external partners to share information on APRI policy research findings and conclusions; and
provide training and support to help Agency personnel better navigate the Agency’s Library, which includes APRI research studies.

APRI Management supported by ACOA Library, the PRWG and the DG PAC.

April 2016

Ongoing

3. In tandem with current PAC efforts, calibrate the performance measurement approach to better inform the APRI program and Agency decision making by identifying, tracking and analyzing key indicators that focus on monitoring both program outputs and immediate outcomes. (Performance Measurement)

Agree. An up-to-date, calibrated PMS that integrates APRI PM requirements within the overall PAC strategy will help to strengthen PM practices for APRI and for PAC.

DG PAC is developing a revised PMS for PAC that incorporates the APRI. Given APRI’s low risk and materiality, this new strategy will be calibrated in such a way that the key performance indicators and expected results associated with APRI’s key activities (i.e. research studies, engagement and networking events) will support results-based management practices especially as they relate to program outputs and immediate outcomes. APRI Management will also work with proponents so they are aware of the expected results and key indicators so that project reporting mechanisms are in place to support this new strategy.

DG PAC supported by APRI team.

April 2016

Appendix B – Evaluation questions, judgment criteria and methods

When judgment criteria in the table below refer to the results of previous evaluations as a benchmark for assessing current program success, it should be noted that in some cases, baseline information is not available. In those cases, this evaluation will be used to gather baseline data for future evaluations.

Evaluation Questions	Judgment Criteria	Method					Indicators
		Literature/Document Review	Interviews	Focus Groups	Administrative Data	Case Studies
Relevance – extent to which APRI addresses a demonstrable need and is relevant to ACOA’s mandate and strategic objectives as well as government-wide priorities and strategies.
Issue 1: Continued Need for the Program
What is the current situation in Atlantic Canada with respect to the need for economic policy research and engagement?	The needs that the programming is expected to meet are still present to at least the same degree as they were five years ago.	X	X	X			Evidence of current or emerging needs for economic policy research in Atlantic Canada
1.2 To what extent are the needs of stakeholders being met? To what extent, and how, has APRI been responsive to existing and emerging policy research needs?	As appropriate, current and planned program activities proactively address changes in context. Evidence of and views on activities and outcomes related to changes in the scientific, social and/or political context in which the program operates. Evidence of and views on best practices and lessons learned that are applicable to current regulatory initiatives.	X	X	X		X	Program activities and reach address current needs Views of stakeholders on program connection to needs
Issue 2: Alignment with Government Priorities
2.1 To what extent is APRI aligned with federal government and ACOA roles and priorities relating to economic policy research and engagement in Atlantic Canada?	There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome. The alignment is recognized, communicated and/or made explicit.	X	X	X	X	X	Evidence of alignment with: federal priorities and strategies departmental strategic outcomes
Issue 3: Alignment with Federal Roles and Responsibilities
3.1 To what extent, and how, do other policy research funding programs or mechanisms complement, overlap or duplicate the objectives of the APRI within ACOA and/or at the federal, provincial or local levels (e.g. academic, industry, etc.)?	ACOA is mandated by law to fulfill the role. Other jurisdictions administer such programming through the federal government. ACOA’s program complements rather than duplicates or overlaps other federal/provincial programming. Needs are coordinated with alternative services or funding programs within ACOA (extent of complementarity, duplication or overlap). Opportunities for improved coordination are being acted upon.	X	X	X		X	Evidence of alignment with federal activities, roles and responsibilities Evidence of complementarity, duplication or overlap with other government programs or private-sector services
Performance
Issue 4: Effectiveness – the extent to which APRI objectives have been achieved within the context of expected results and outcomes.
4.1 How well is APRI achieving its pan-Atlantic program objectives? Does APRI contribute to the immediate outcomes of PAC? If so, to what extent and in what ways has APRI contributed to: Policy: well-informed policy decisions reflecting opportunities and challenges of the Atlantic region’s economy while considering enterprise and community development potential. Advocacy: Atlantic enterprise and community development interests being considered in emerging and changing federal economic policies, programs and regulations. Coordination: the coordination of partners in addressing the economic priorities of Atlantic Canada through a coherent approach to development. * Used 2014-2015 ACOA PMF for outcomes for PAC (a, b, c).	APRI is fulfilling the pan-Atlantic aspect of its mandate to undertake activities that have an Atlantic Canada scope (i.e. impact more than one Atlantic province). There is sufficient qualitative and/or quantitative evidence to argue the contribution of the APRI to the achievement of the immediate outcomes of PAC. The achievement of outcomes is similar to or greater than that observed in the previous evaluation (accounting for changes in context, processes and procedures).	X	X	X	X	X	Evidence in the nature of projects funded and networks established that the pan-Atlantic mandate of APRI is being achieved Evidence that a strong alignment exists between the results achieved by APRI and PAC immediate outcomes Evidence of (performance data on) extent to which, and ways in which, APRI has contributed to a, b and c Stakeholder views on achievement of this outcome
4.2 In what ways has policy research been made available to decision-makers in a timely manner to act on economic issues?	Stakeholder perceptions indicate that decision-makers received policy research outputs in a timely manner to support decision making on economic issues		X	X		X	Stakeholder perceptions on the achievement of this outcome
4.3 To what extent, and in what ways, do facilitative activities (e.g. engagement, outreach, relationship building, exploration, consensus building) support the achievement of broader program objectives such as the promotion of forward thinking around key, evolving and sometimes higher risk issues?	There is sufficient qualitative evidence (including examples) to argue that the APRI program has contributed to advancing forward thinking with respect to economic policy research capacity in Atlantic Canada.		X	X		X	Evidence of the extent to which, and ways in which, APRI has contributed to advancing forward thinking with respect to economic policy research capacity in Atlantic Canada Stakeholder views on achievement of this outcome
4.4 To what extent, and in what ways, has there been two-way communication (i.e. reciprocal information sharing) between the APRI program and both internal and external stakeholders (including program staff and regional offices) to support: awareness of APRI and stakeholder objectives; engagement in terms of priority setting; and dissemination and use of outputs? * Informed by 2010 APRI evaluation and 2012 update of APRI MAP	There is sufficient qualitative and/or quantitate evidence to argue that internal and external communication activities have been effective. The communications activities, outputs and outcomes have been enhanced since they were observed in the previous evaluation (accounting for changes in context, processes and procedures).		X	X		X	Evidence of (performance data on) communication activities, outputs and outcomes (number, nature and type of communications including: meetings, reports, correspondence, briefings, etc.) Stakeholder views on, and satisfaction with, communication activities, outputs and outcomes
4.5 To what extent, and how, can social media and other technology be used to engage with stakeholders and support communication and dissemination of policy research findings and conclusions?	Social media and other technology has been used to support the effective communication and dissemination of policy research findings and conclusions.	X	X	X		X	Evidence of (performance data on) effective use of social media and other technology Stakeholder views on, and satisfaction with, use of social media and other technology
Issue 5: Efficiency and Economy – the extent to which activities are undertaken in an affordable manner, taking into consideration the relationship between outputs and the resources to produce them, and the extent to which resources allocated to the APRI are well-utilized, taking into consideration alternative delivery mechanisms.
How effective are the mechanisms and structures within ACOA that facilitate policy research governance (e.g. planning, conducting, dissemination and use)? To what extent is the Agency making optimal use of the knowledge assets generated through policy research? What are the barriers to this optimization?	Qualitative evidence that ACOA has structure/mechanisms in place to ensure that the most efficient and economical means is being used to administer the programming. Governance processes are appropriate: there is a clear delineation of roles and responsibilities; and planning and decision-making processes are clear and streamlined. There is qualitative evidence that the knowledge assets generated through policy research are being optimized (shared, accessible, etc.). There is evidence that factors (internal and/or external) that facilitate or impede the success of the APRI are known, and mitigation strategies that are implemented are appropriate.		X	X	X		Evidence of, and views on, efficiency and economy of current APRI program administration structures and mechanisms Evidence that factors that are impeding the success of the APRI are known and that mitigation strategies exist
5.2 Is there a more cost-effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned? What can APRI learn from other economic policy funding mechanisms in other regional development agencies or similar organizations?	Delivery costs compare favourably to the delivery costs as calculated during the previous evaluation and to the delivery costs of similar programming offered in other jurisdictions. Lessons learned and best practices are identified regularly. There is evidence that program management has considered and continues to explore alternative modes of delivery.		X	X	X	X	Evidence of and views on alternative program models that would achieve outcomes at lower costs (where available) Views on whether funds are appropriately targeted
5.3 Is there appropriate performance measurement in place? If not, what steps are required to support the development and implementation of more effective performance measurement activities? If yes, is performance measurement information being used to inform senior management decisions?	Performance data is being collected and is available for use during the evaluation. Qualitative evidence that performance measurement data is adequate and is effective in reporting on the achievement of outcomes. Qualitative evidence that performance measurement data is considered useful and is being used in decision making.		X	X	X		Existence of performance measurement framework or strategy (including a logic model) Adequate collection of performance information Use of performance information in decision making

Appendix C – ACOA’s Program Alignment Architecture 2014-2015

Strategic Outcome	Programs	Sub-programs
A competitive Atlantic Canadian economy (1.0)	Enterprise Development (1.1)	Innovation and Commercialization (1.1.1) Productivity and Growth (1.1.2) International Business Development (1.1.3)
	Community Development (1.2)	Community Investment (1.2.1) Community-based Business Development (1.2.2)
	Policy, Advocacy and Coordination (1.3)	Policy (1.3.1) Advocacy (1.3.2) Coordination (1.3.3)
	Internal Services (1.4)	Governance and Management Support (1.4.1) Resource Management Services (1.4.2) Asset Management Services (1.4.3)

Appendix D – Logic model cross walk – PAC and APRI

	PAC LOGIC MODEL (2012)			APRI LOGIC MODEL (2010)	CROSS CUTTING THEMES
	Policy	Advocacy	Coordination	APRI LOGIC MODEL (2010)	CROSS CUTTING THEMES
Activities/ Outputs	Analysis and focused research of policy issues, trends, challenges, opportunities, best practices and emerging approaches related to regional economic development Macroeconomic, microeconomic and fiscal analyses Engagement of research partners and stakeholders on Atlantic regional economic development issues and priorities	Increased awareness and capacity building in area of strategic industrial interest, such as aerospace and defense Involvement in the federal policy making process	Federal/provincial initiatives (number and type of formal and informal initiatives) Coordinated and shared initiatives with other federal departments via the federal Regional Councils Partnership and network mechanisms that demonstrate a strong federal presence with regional and national stakeholders	ACTIVITIES Coordinate and plan appropriate research and engagement activities Select appropriate research and engagement activities Fund research and engagement activities Develop networks of policy stakeholders Disseminate research and engagement information OUTPUTS Criteria for identifying relevant research topics and programs and/or projects Contributions, contracts and funds leveraged Research projects and reports with recommendations for policy and program development or refinement, or for future research efforts Roundtables and conferences Advice to ACOA senior management and/or minister Process map for research and engagement Policy research education partnerships Partnerships formed for joint activities Disseminated reports and materials	Conducting research leads to: focused research on policy issues, trends, challenges, etc. of the Atlantic regional economy Undertaking engagement leads to: better coordinated networks of research partners and stakeholders with common interests Communicating research and undertaking engagement activities lead to: knowledge and increased awareness about opportunities and challenges ability to be involved in, coordinate and influence both regional and national policies, programs and regulations enhanced policy capacity internally and externally
Expected Results/Key Outcomes	Well informed policy decisions reflecting opportunities and challenges of the Atlantic Region’s economy while considering enterprise and community development potential	Atlantic enterprise and community development interests are reflected in emerging and changing federal economic policies, programs and regulations	Coordination of partners in addressing the economic priorities of Atlantic Canada through a coherent approach to development	IMMEDIATE OUTCOMES Better understanding of the regional economy, strategic sectors or areas for support, and appropriate directions for action in identifying opportunities for economic growth in the Atlantic region Accumulated knowledge of issues and challenges in Atlantic Canada, for use in influencing regional and national positions of federal departments and agencies and other stakeholders, and to prepare better to address Atlantic issues more effectively Building and maintaining relationships, more focused and better coordinated network of research partners and stakeholders with common interests
Program Expected Results/Key Outcome	Policies and programs that strengthen the Atlantic economy			INTERMEDIATE OUTCOMES Consideration of recommendations in policy or program design or for further research events Contributions to, or influence on, ACOA’s strategic priorities, and more coherent process for strategic plan formulation and renewal Integration of findings from engagements or research into policy papers, advice to ministers or Cabinet Enhanced policy research capacity in the region that is recognized and used by other stakeholders
Strategic Outcome	A competitive Atlantic Canadian economy			ULTIMATE OUTCOME: Contribution to an increased enterprise and community development in Atlantic Canada
Source	Evaluation of the Atlantic Canada Opportunities Agency Policy, Advocacy and Coordination Activity (March 6, 2012)			Evaluation of the Atlantic Canada Opportunities Agency’s Atlantic Policy Research Initiative (March 17, 2010)

Footnotes

[i] Efforts are under way to develop a current PAC logic model and the PMS in early 2015 that will consider the contributions of APRI to the overall PAC function and that may include a stand-alone APRI logic model (see 2.2 Program Theory).

[ii] DG PAC Committee Terms of Reference, April 2013.

[iii] Policy Research Working Group Terms of Reference, June 2013.

[iv] Harrison and Sharp. “The productivity performance of Atlantic Canada,” Centre for the Study of Living Standards, a study funded by ACOA, 2009.

[v] Thinking about knowledge mobilization, the Canadian Council on Learning and the Social Sciences and Humanities Research Council, 2008.

[vi] Building Policy Research Capacity, The Conference Board of Canada, 2007

[vii] Building Policy Research Capacity, Conference Board of Canada, 2007.

[viii] Thinking about knowledge mobilization, the Canadian Council on Learning and the Social Sciences and Humanities Research Council, 2008.

[ix] According to the Treasury Board Secretariat Directive on the Evaluation Function (Canada, 2009a), demonstration of efficiency and economy is an “assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.”

[x] Institute on Governance. “Defining Governance”

Page details

Date modified:: 2020-04-23

Evaluation of the Atlantic Canada Opportunities Agency Atlantic Policy Research Initiative 2009-10 to 2013-14

Glossary of acronyms

Table of contents

List of figures

List of tables

Acknowledgements

Executive summary

1.0 Introduction

1.1 Evaluation overview

1.2 Evaluation design and methodology

1.3 Evaluation strengths, limitations and mitigating measures

2.0 Profile of the Atlantic Policy Research Initiative

2.1 Context

2.2 Program theory

2.3 Program accountability and governance

2.4 Expenditure profile

2.4.1 Project profile

3.0 Findings: relevance

3.1 Continued need for the programming

3.2 Alignment with government priorities

3.3 Alignment with federal roles and responsibilities

4.0 Findings: performance – effectiveness

4.1 Achievement of expected results

4.2 Timeliness of policy research

4.3 Value of facilitative activities

4.4 Approaches to reciprocal information sharing

4.5 Use of social media and other technology

5.0 Findings: performance – efficiency and economy

5.1 Efficient utilization of resources

5.1.1 Structures and mechanisms that support efficiency and economy

5.2 Alternative modes of delivery, lessons learned and best practices

5.2.1 Alternative modes of delivery

5.2.2 Best practices generated from the evaluation

5.3 Adequacy of performance measurement

6.0 Conclusions and recommendations

6.1 Conclusions

6.2 Recommendations

Appendix A – APRI evaluation management action plan

Appendix B – Evaluation questions, judgment criteria and methods

Appendix C – ACOA’s Program Alignment Architecture 2014-2015

Appendix D – Logic model cross walk – PAC and APRI

Footnotes

Page details