This evaluation was undertaken to provide the Atlantic Canada Opportunities Agency (ACOA) management with systematic, neutral evidence on the relevance and performance of its Community Mobilization (CM) and Community Investment (CI) sub-programs. It is expected that the results of this study will be used to support the future direction of programming. The study was managed and completed by ACOA’s Evaluation Unit.

We would like to take this opportunity to thank the members of the evaluation advisory committee and the evaluation working group for their advice and support throughout this process. Their assistance helped to ensure the relevance and usefulness of this evaluation. Of note is the gracious contribution of the director general for community development, the regional program directors and managers, and three external members: Dr. Kelly Vodden, Memorial University of Newfoundland; Lise Ouellette, Community College of New Brunswick; and Daniel Fairbarn, Industry Canada.

We are also grateful to the many ACOA staff members, including staff from the Evaluation Unit, as well as ACOA clients and external key informants who provided their time and essential knowledge in support of this study.

Overall, these contributions were instrumental in building our understanding and correctly conveying the nature of the community development planning and investment programming at ACOA.

Courtney Amo Acting Director and Head of Evaluation, Evaluation and Risk Directorate Atlantic Canada Opportunities Agency

Executive Summary

ACOA’s Community Mobilization (CM) and Community Investment (CI) sub-programs are part of the Agency’s Community Development (CD) program. Through its CD program, ACOA works with communities and entrepreneurs to stimulate economic growth, improve infrastructure and develop opportunities in local economies, with the desired outcome of creating dynamic and sustainable Atlantic communities.

In support of this outcome, the CM sub-program supports the mobilization of key stakeholders for the planning of economic development initiatives, while the CI sub-program invests in critical economic development infrastructure and other initiatives that support economic development outcomes.

CM and CI programming is delivered using a decentralized model, involving all ACOA regional and district offices working in close partnership with a variety of government and non-governmental stakeholders. Two main transfer payment programs are used: The Innovative Communities Fund (ICF) and the business support element of the Business Development Program (BDP). Overall, an average of approximately $103 million per year is expended through CM and CI programming, representing 28 per cent of ACOA’s total expenditures.

The purpose of this evaluation is to assess the relevance and performance of CM and CI programming and to fulfill Government of Canada accountability requirements. The evaluation covers the four-year period from fiscal year 2008-2009 to 2011-2012. Although CM and CI are separate sub-programs, they have been evaluated together to acknowledge the complementarity between economic development planning and investments and to address challenges in how the sub-programs are differentiated in administrative data. This is in keeping with the terms of reference for the study, which were approved by ACOA’s executive committee in June 2012.

The evaluation methodology included 13 case studies, 71 key informant interviews, a client survey with an overall response rate of 49 per cent, a document and literature review, and a detailed analysis and categorization of over 1,000 projects. Close to 300 individuals provided their knowledge and insight to this study. Given that operational funding to regional economic development organizations (REDOs) across Atlantic Canada was ceased as of May 2013, only existing data related to REDOs was considered in this evaluation. Evaluation findings are based on a high level of convergence of multiple lines of evidence and are deemed reliable and valid within the context of the study limitations.

Findings and Conclusions

Relevance

The CM and CI sub-programs are relevant activities for ACOA and the federal government. The programming aligns with ACOA’s strategic outcome and the ACOA Act, and activities are consistent with the Agency’s roles and responsibilities and broader federal priorities.

There is a continued need for planning and investment programming as challenges to community economic development (CED) in Atlantic Canada exist to the same degree or are greater than reported in the previous evaluation. In particular, the evaluation reports a substantial increase (around 30 per cent) since the last evaluation in the number of clients who cite challenges related to the aging population and outmigration as key barriers to economic development.

In response to changing needs, the Agency has adjusted its programming. ACOA has endeavoured to invest in projects that generate greater economic impacts (including economic growth and job creation) and has discontinued operational funding to REDOs across Atlantic Canada in order to reduce duplication of services, to streamline delivery and to acknowledge the changing demographics of rural areas.

ACOA’s CM and CI programming is unique and complementary to CED programming offered by other organizations, and is similar to what is offered through other regional development agencies (RDAs). The programming’s distinctive traits include: its Atlantic-wide scope; the contributions of ACOA’s Policy, Advocacy and Coordination (PAC) function; internal capacity for proposal development and assessment; programming flexibility; and focus on CED goals.

A collaborative, coordinated and strategic approach among partners is critical to the achievement of CED outcomes. ACOA works with, and alongside, a variety of partners and stakeholders toward these outcomes. The dissolution of the REDO model brings about a necessary period of transition as ACOA and CED partners realign resources and clarify roles and responsibilities. During the transition period, there is a risk of capacity gaps for regional-level planning or other economic development activities.

Performance

Incrementality

CM and CI programming is incremental to the implementation of projects, to obtaining investments from other partners and to achieving results. Almost two thirds (64 per cent) of surveyed clients reported that their projects would likely not have proceeded at all without ACOA’s support; and of those projects that could have proceeded, all but 3 per cent report negative impacts, including smaller scope, time delays, poorer quality and difficulty securing other funding. Over two thirds (70 per cent) of surveyed clients indicated that other funding was secured as a result of ACOA’s decision to provide support. On average, clients report that 42 per cent of total project funding was secured from external sources as a result of ACOA. These results were echoed by key informants, who reported on the influence played by ACOA’s investment decision.

Achievement of Expected Outcomes

Overall, the evaluation has found that the majority of both planning and investment projects achieved the immediate results expected of CM and CI programming.

There were four main types of planning projects over the evaluation period: 1) support for REDO operations, including core planning activities; 2) other regional economic development planning; 3) industry sector development planning; and 4) studies or plans related to specific projects or initiatives.

In general, planning projects resulted in partner mobilization, capacity building and the development of concrete plans, which supported increased CED skills and leadership, decision-making influence and concrete actions for economic development. Due to low survey representation of projects with completed plans and strategies, it is not possible to develop conclusions on the overall achievement of intermediate outcomes of planning activities. However, survey findings and case studies support the importance of thorough project planning for greater economic development impacts.

Over the evaluation period, the largest category of investment projects was related to construction of infrastructure (61 per cent of funding). Within this category, 55 per cent of funding supported community, recreational and cultural infrastructure. These projects and their expected outcomes are not well articulated in CI programming documents. In addition, project analysis revealed that a large proportion of investment funding was linked to tourism, either by supporting ACOA’s Growth Strategy for Tourism (35 per cent or $49.7 million), or by having other tourism implications (27 per cent or $38.3 million). The extent to which CI programming aims to target the tourism sector and its importance in relation to other types of projects is not clear in CI documentation.

Beyond the funding that supported tourism and community infrastructure for recreation and culture, a combined 35 per cent ($64.5 million) of funding supported: infrastructure and equipment purchases for industry/sector-related training, research and other activities; infrastructure that enables economic activity such as broadband, sewage/drainage and electrical upgrades; buildings that accommodate commercial space; and training to enhance business or industry-related skills.

Investment projects generally achieved immediate outcomes such as creation of infrastructure for economic development and increased capacity for responding to future opportunities. Intermediate outcomes included increased tourism visitation and sales and revenues for businesses, as well as the development of new products. There was evidence of modest job creation and maintenance as well as modest business creation and survival.

Overall, the evaluation provides evidence that the capacity and asset base of communities is strengthened as a result of planning and investment funding offered by ACOA and its partners. By strengthening their capacity and asset base, communities can better respond to economic opportunities and threats, increase their economic stability and sustainability; and improve the quality of life of residents. However, the evaluation cannot speak to the overall impact of this strengthened asset base on the economic viability and sustainability of communities.

Barriers to achieving results exist, and could be better mitigated through improved project viability/risk assessment processes and the adoption of promising practices tested in some regions. In addition, the extent of coordination of projects that cut across ACOA programming areas (e.g. business skills development, tourism, trade, innovation) was unclear, which may create inefficiencies in achieving Agency objectives relating to those areas. Limited information on activities, outputs and outcomes for individual projects and at the aggregate level presents challenges to this type of coordination and to the strategic, results-based management of CM and CI programming.

Overall, the CM and CI programming was delivered in a cost-effective manner. Internal costs were proportionate to the amount of funding delivered and compared reasonably between ACOA regions. ACOA support enabled projects to leverage substantial funds from other organizations. Leveraging of other funding has increased since the previous evaluation. Mechanisms that fostered efficient and economical delivery included enhanced planning processes, efforts to achieve greater economic impacts, collaboration among ACOA units, and coordination with other RDAs in the delivery of national initiatives. ACOA was efficient in delivering substantial EAP programming without additional dedicated resources; however, this resulted in fewer projects related to ACOA’s core CM and CI programming being delivered.

Recommendations

The evaluation team has issued three recommendations, which are summarized below and further detailed in the report. The recommendations are aimed at improving programming by balancing a consistent, targeted approach with required flexibility; improving the availability and access of information and knowledge to manage for results; and building on existing and new relationships, expertise and intelligence to narrow the gap between ACOA’s community economic development theory and practice, as implemented through the CM and CI sub-programs.

Recommendation 1

In order to ensure a clear corporate direction for programming while allowing for regional variability and flexibility, ACOA should build on its corporate knowledge, lessons learned and current best practices in CED to formulate a Community Investment framework. The framework should outline the Agency’s approach to planning and investment programming (including main project types and expected outcomes), while acknowledging regional context, plans, priorities and best practices. It should also describe the Agency’s approach to engage, coordinate and collaborate both internally and externally (i.e. with partners and subject matter experts) in the achievement of programming outcomes.

Recommendation 2

In support of results-based management, decision making, and the strategic coordination of initiatives that cut across programming areas (e.g. business skills development, tourism, trade, innovation), ACOA should ensure that quality and timely information on activities (i.e. main project types) and outcomes are systematically collected on file and in electronic systems.

Recommendation 3

In an effort to increase efficiency and effectiveness of programming and mitigate risks of not achieving expected outcomes, ACOA should build on current efforts to develop and systematically implement a standardized process/tool to assess project viability. The process/tool should assess the likelihood of achieving outcomes as well as the human, financial, and infrastructure capacity of the proponent organization or community to conduct the work outlined in the proposal and to sustain the achievement of outcomes and impacts following project completion.

Management has agreed with the evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the evaluation’s recommendations, can be found in Appendix D.

1. Introduction

This report presents the findings, conclusions and recommendations of the evaluation of ACOA’s Community Mobilization (CM) and Community Investment (CI) sub-programs. The programming was evaluated for relevance and performance in accordance with the Treasury Board (TB) Policy on Evaluation, the Agency’s approved evaluation plan, and the approved terms of reference for this study.

The CM and CI sub-programs support the achievement of ACOA’s Community Development (CD) program goal to foster dynamic and sustainable Atlantic communities and more broadly, the Agency’s mandate “to increase opportunity for economic development in Atlantic Canada” [1]. CM programming focuses on mobilizing stakeholders and planning for regional or community economic development initiatives. CI programming invests in critical economic development infrastructure and other initiatives that support economic development outcomes.

The evaluation examined a four-year period, from 2008-2009 to 2011-2012 [I] . The scope of the evaluation excludes measuring outcomes achievement for programming related to Canada’s Roadmap for Linguistic Duality and the Economic Action Program (EAP) as they were reported on separately. However, the evaluation considered best practices and lessons learned, particularly from the delivery of EAP programming, to inform future delivery of similar short-term initiatives. Also, these initiatives were considered as part of the programming delivered by CI staff in the evaluation’s assessment of CM and CI efficiency and economy (Section 5). [II]

Following the evaluation overview presented below, Section 2 of the report provides a profile of the programming. Sections 3 through 5 present the study’s findings, organized by broad evaluation question (relevance, performance-effectiveness, performance-efficiency and economy), and Section 6 presents the conclusions and recommendations resulting from the study.

Evaluation Overview

The purpose of this evaluation was twofold. First, to provide timely, credible and neutral information on the relevance and performance of CM and CI programming in order to support decision-making, continuous improvement and results-based management. Second, to fulfill accountability requirements set forth by the TB Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada [2].

The research was completed between June 2012 and June 2013. An extensive planning phase that examined all sub-programs within the CD program ran from August 2011 to June 2012. During this phase, it was determined that the CM and CI sub- programs should be evaluated together given the complementarity between economic development planning and investments, and inconsistencies in the coding of planning and investment projects to each sub-program within ACOA’s administrative database. To ensure that projects were considered appropriately in the evaluation, the evaluation team conducted a detailed review of all projects to identify the nature of the activities under the two broad categories of planning or investments. Therefore, this report refers to the overall set of projects considered in the evaluation as economic development “planning and investment programming.”

ACOA’s Evaluation Unit planned and conducted the evaluation with the assistance of consulting firms for two case studies and the client survey [III]. An evaluation advisory committee (EAC), with representatives from ACOA management and staff as well as external stakeholders with specialized knowledge in community economic development (CED), provided advice and guidance throughout the evaluation. A working group of managers from all ACOA regions and Enterprise Cape Breton Corporation (ECBC) provided access to information and other resources. The EAC and working group helped ensure clear understanding of the programming and supported the interpretation of findings, which added to the relevance and usefulness of the final report and its recommendations.

1.2 Evaluation Design and Methodology

The evaluation of the CM and CI sub-programs addresses five core issues that fall into two broad categories, relevance and performance, in accordance with the TB Policy on Evaluation. Table 1 identifies the specific evaluation questions for each core issue. These were developed during the planning phase that preceded the evaluation, when key stakeholders were able to raise possible issues for further study.

Table 1: Evaluation Questions by Core Issue

Issue	Evaluation Question
Relevance
Issue 1: Continued Need for the Program
1.1	To what extent do the CM/ CI sub-programs continue to address a demonstrable need?
1.2	To what extent are the CM/ CI sub-programs responsive to the existing and emerging needs of Canadians? What other mechanisms exist to address these needs?
Issue 2: Alignment with Government Priorities
2.1	To what extent are the CM/ CI sub-programs aligned with federal government priorities and expectations, and to ACOA’s strategic outcome?
Issue 3: Alignment with Federal Roles and Responsibilities
3.1	To what extent do the CM/ CI sub-programs align with federal roles and responsibilities?
Performance
Issue 4: Effectiveness
4.1	How and to what extent are the CM/ CI sub- programs achieving expected results (immediate and intermediate outcomes)? What impact would the absence of the CM/ CI programming have on projects/initiatives? What are the barriers to the achievement of expected outcomes, and to what extent have these been mitigated? What lessons have been learned, and how can these contribute to future programming? What unintended outcomes have been achieved? To what extent are the CM/ CI sub- programs’ performance measurement and reporting structures effective in reporting on the achievement of outcomes? How is the information used by ACOA and ACOA-funded organizations?
Issue 5: Efficiency and Economy
5.1	In the context of the results being achieved, to what extent are the resources allocated to the CM and CI sub-programs efficiently utilized?
5.2	Is there a more efficient and effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned?

Judgment criteria, or “benchmarks” for successful achievement, were developed with advice from program management for each evaluation question [3] . The evaluation team used the judgment criteria to objectively and transparently assess the relevance and performance of the programming. The judgment criteria were considered in determining data needs, methodology and analysis. [4]

A risk-based approach was used to design the evaluation and methods, as required by the TB Policy on Evaluation. The level of effort associated with conducting the evaluation was calibrated to reflect risks associated with the relevance, effectiveness and efficiency/economy of the programming, as determined through a document review and stakeholder consultations. Considering evaluation issues, risk and judgment criteria, a mixed methods research design was chosen involving multiple lines of evidence gathered through the following methods:

Case studies – Completion and analysis of 13 case studies representing different types of projects and ranges of project success. Due to the complex nature of the programming and its context, case studies were critical and central to the evaluation team’s understanding of how and why activities lead to outcomes, as well as key challenges and facilitating factors. They helped to contextualize other lines of evidence and greatly supported the development of conclusions and recommendations. Case studies used document review and interviews with internal and external stakeholders, and were validated through consultations with key informants. Clients were asked whether their identities could be shared as part of case study examples. This was done only in cases where permission was granted by the client.
Administrative data review – Analysis of project data from ACOA’s administrative and financial databases, including the detailed analysis and categorization of over 1,000 projects
Document and literature review – Comprehensive review of relevant internal and external documents, including scholarly and grey literature
Interviews – In-depth interviews with 71 key informants (including those completed within case studies), including a cross-section of internal and external stakeholders
Client survey – Analysis of data from a web-based survey for planning and investment projects, representing a 49 per cent (228 out of 465) overall response rate; 77 per cent (41 out of 50) for planning projects and 45 per cent (187 out of 415) for investment projects. [IV]

See Appendix B for further detail on the evaluation design, calibration of effort, methods, and judgment criteria.

Note that the evaluation team did not collect new performance information from regional economic development organizations (REDOs) through the client survey or key informant interviews due to ACOA’s cessation of operational funding during the evaluation process (see section 2.1 for further information). However, existing performance measurement data related to the REDOs were considered in assessing outcome achievement.

1.3 Evaluation Limitations, Mitigating Measures and Strengths

The evaluation design and implementation are considered appropriate based on the intended objectives of the study. Multiple lines of evidence gathered through a mix of qualitative and quantitative methods allowed for triangulation (i.e. a convergence of results across lines of evidence) and complementarity of findings (i.e. developing better understanding by exploring different facets of a complex issue). The diversity of methods helped compensate for the inherent limitations of each data source and helped mitigate the overall study challenges.

Study challenges included programming complexity, programming changes during the evaluation period, and the limited availability and reliability of data on the funded activities and their performance. While these data were subsequently collected through a client survey, they are subject to the limitations of the survey method and to a reduction in data validity due to the need to recreate data on past performance. Also, since REDOs were not included in the client survey, only a small number of completed planning projects are represented, which limits the assessment of intermediate planning outcomes.

Mitigation measures included the elaboration of program theory to better understand program complexity, the use of case studies representing a range of project success, the contextualization of program changes, a detailed categorization of project types to create a profile of the projects/activities funded, and the development of methods that address gaps with respect to the availability and reliability of performance information.

Strengths of the evaluation design included consideration of the broader program theory, the involvement of key stakeholders during methodology design, and the high client survey response rate (49 per cent overall).

Across methods, close to 300 individuals provided their knowledge and insight to this study, and the findings presented in the report build on a high level of convergence of informed views and opinions. Within this context, the results of the CM and CI sub-program evaluation are deemed reliable and valid. See Appendix B for further detail on the evaluation challenges, mitigation strategies and strengths.

2. Profile of the Community Mobilization and Community Investment Sub- programs

2.1 Context

Community Development (CD) is one of four programs within ACOA’s Program Alignment Architecture (PAA) (Appendix A) that contribute to achieving the Agency’s strategic outcome of a competitive Atlantic Canadian economy. Through its CD programming, ACOA works with communities and entrepreneurs to stimulate economic growth, improve infrastructure and develop opportunities in local economies, with the desired outcome of creating dynamic and sustainable Atlantic communities. Rural communities are a particular priority of the programming.

CM and CI are two of the four sub- programs that support the goals of the CD program. CM programming supports the mobilization of key stakeholders for planning of economic development initiatives, while CI programming invests in critical economic development infrastructure and other initiatives that support economic development outcomes.

CM and CI programming are delivered using mainly two transfer payment programs: The Innovative Communities Fund (ICF) and the business support element of the Business Development Program (BDP).

The objectives of the ICF are to enhance community or regional capacity through the development of competitive, productive, strategic industry sectors; to strengthen community infrastructure in rural communities to improve their economic development capacity; and to support initiatives that enhance communities’ capacity to overcome economic development challenges and take advantage of their strengths, assets and the opportunities presented. [5]

The objectives of the BDP business support element are to strategically assist not-for-profit organizations to provide specialized services and infrastructure in support of entrepreneurship and small and medium-sized enterprises (SMEs), or to assist local economic development that further advances the Agency’s strategic priorities and the federal government’s regional development policies. [6]

CM and CI programming also support broader ACOA areas of focus such as:

responding to the economic development needs of Indigenous communities and Official Language Minority Communities (OLMCs);
supporting ACOA’s Growth Strategy for Tourism, which seeks to maximize the tourism industry’s impact on the Atlantic economy through strategic tourism product development and marketing, research and planning, and the development of skills/capacity of tourism operators; and
supporting the development of business skills across Atlantic Canadian communities.

One important recent change to CM programming is the cessation of operational funding to 51 REDOs across Atlantic Canada as of May 2013. Over the evaluation period, REDOs received a substantial amount of CM funding ($29 million) in support of their operational costs, which funded their core regional economic development planning and mobilization activities.

The decision to discontinue REDO operational funding was based on the need to reduce duplication of services, to streamline delivery by ensuring a more direct service to communities and other key clients, and to increase focus on investments that maximize economic growth and job creation. This decision was, at least in part, based on a previous evaluation of the programming that found evidence of overlap between REDO activities and other organizations in some regions. [7] Also, the appropriateness of the REDO model was considered due to the extent of the demographic decline experienced in some rural areas served by REDOs.

2.2 Special Initiatives

Over the evaluation period, CM and CI staff also delivered the Atlantic portion of Canada’s Economic Action Plan (EAP) initiatives: the Recreation Infrastructure Canada program (RInC) and the Community Adjustment Fund (CAF). These initiatives aimed to create employment opportunities and address issues of declining infrastructure in communities affected by the global economic downturn. [8]

CM and CI staff also delivered projects in support of Canada’s Roadmap for Linguistic Duality. Specifically, the Economic Development Initiative (EDI) aimed to facilitate sustainable growth in OLMCs, and the New Brunswick Francophone Immigration Project aimed to increase the number of francophone immigrants as well as their social and economic integration into the province. Further detail on the number of projects and amount of funding delivered under special initiatives is contained in section 2.6.

2.3 Program Theory

In general, programs are purposeful and organized efforts to intervene in an existing process or situation in order to provide a service or solve a problem [9]. A program’s “theory” is the explanation of how and why a given intervention is supposed to work, taking into consideration the assumptions, causal mechanisms and other factors that contribute to the achievement of expected outcomes [10]. In essence, it explains how a program, within the context of the environment in which it operates, is expected to produce results [11].

2.3.1 Program Logic Model

A logic model – which sequences the activities, outputs and expected outcomes of a program – is often used to summarize a program’s theory. The logic models for the CM and CI sub- programs are summarized in Table 2.

Table 2: Community Mobilization and Community Investment Logic Models

Component	Community Mobilization	Community Investment
Activities/Outputs	Consultations Information sessions Maintain/create partnerships Planning Financial support (from ACOA) Coordination	Consultations Networking opportunities Information sharing Investments in support of community projects
Reach	Clients: REDOs; NGOs; non- commercial organizations; local governments; OLMCs; Indigenous communities Stakeholders: governments; communities; private sector	Clients: geographic communities; sectoral communities (industry sectors); communities of interest (Indigenous, francophone) Stakeholders: community leaders; financial institutions; private- sector organizations; educational institutions
Immediate Outcomes	Sector development strategies Regional-based strategic plans Increased leadership capacity in communities Improved community collaboration	Increased ability of communities to respond to opportunities and economic crisis Increased economic opportunities Community based partnerships Development of critical economic development infrastructure, including non-physical infrastructure
Intermediate Outcomes *Impacts [V]*	Improved community capacity to identify economic development needs and opportunities The implementation of plans and initiatives that have led to economic impacts	Improved capacity to address economic and business development needs and opportunities Economic activities within businesses, communities and industries The creation of economic opportunities that have led to economic outcomes in the community
CD Outcome	Dynamic and sustainable communities for Atlantic Canada
ACOA Strategic Outcome	A competitive Atlantic Canadian economy

Source: Adapted from ACOA’s Performance Measurement Strategy, June 2010.

Broadly speaking, CM planning and mobilization activities were expected to lead to the development of initiatives that result in economic development impacts in the longer term. CI investments in infrastructure and other initiatives were expected to more directly facilitate economic development impacts. Desired impacts were further elaborated through the development of the CED Conceptual/Analytical Framework [VI] (Appendix C), and have been added to the original logic models. The achievement of outcomes and impacts is influenced by factors relating to existing community or industry assets and other contextual elements.

A detailed description of the activities and expected outcomes listed in the logic model is provided below. Further detail on CM and CI inputs and outputs is provided in section 2.5.

Community Mobilization

CM activities focus on the mobilization of key stakeholders and on economic development planning. These activities include support to REDOs as well as to industry sectors and communities (including Indigenous and francophone communities) whose activities encompass social, commercial and economic interests.

CM activities are expected to lead to immediate outcomes such as leadership skills in economic development, increased collaboration for economic development, and sector- or region-based economic development plans. Intermediate outcomes include improved community capacity to identify needs and opportunities and the implementation of initiatives that focus on specific opportunities that can lead to economic activities within businesses, industry sectors and communities, and economic outcomes for communities such as investments in infrastructure, increased revenues for businesses and job creation.

Community Investment

CI activities focus on investments that facilitate economic development in communities, broader regions and specific industry sectors. CI supports critical economic development infrastructure for communities and industries, marketing initiatives, skills capacity and other types of initiatives. ACOA also works with Indigenous and francophone communities to identify investments that can stimulate transformative change and support strategic development. CI funding is also used to help communities in economic adjustment or transition by providing support in areas that help them respond most effectively to their unique situations.

The initiatives funded through CI are expected to lead to immediate outcomes such as greater capacity of communities to respond to opportunities and economic crisis, partnerships for economic development, and critical infrastructure. Intermediate outcomes include improved capacity to address economic and business development needs and opportunities as well as increased economic activity within businesses, industry sectors and communities, and economic outcomes for communities such as increased revenues for businesses, increased tourism visitation, and creating or maintaining jobs and businesses.

2.3.2 Other Influential Factors

Literature and key stakeholders consulted in the development of the CED Conceptual/Analytical Framework (Appendix C) pointed to the importance of considering the role of physical, human, social, monetary and natural assets that contribute to and influence the achievement of outcomes along with ACOA support. Other external factors such as global economic forces, population shifts, and environmental changes can also influence the impact of economic development initiatives. These were explored in case studies and were considered in interpreting evaluation findings. The information was also useful for framing current challenges in community economic development, as they relate to programming relevance and need.

2.4 Program Accountability and Governance

ACOA’s Executive Committee is chaired by the President (Deputy Minister) and is the highest internal governing forum that supports the President in the development of the Agency’s policies and programs. Members of the Executive Committee include ACOA vice presidents (VPs), the Chief Executive Officer (CEO) of Enterprise Cape Breton Corporation (ECBC), and other senior executives. Each regional office is headed by a VP who is responsible for ACOA policy and program activities within their respective region (N.B., P.E.I., N.L., and mainland N.S.). Based in Sydney, N.S., the CEO of ECBC is responsible for the delivery of ACOA programming within Cape Breton, in addition to the Corporation’s own programs.

The Director General (DG) of CD, located at Head Office in Moncton, N.B., together with regional directors of CD, is responsible for CD programming across the Agency. The DG of CD reports directly to the VP of Policy and Programs, while regional directors of CD report to the DG of Operations and the VP of their respective regions. ECBC has a DG of CD who reports internally within the Corporation.

With respect to approval of funding under the ICF transfer payment program, the minister provides direction to the regional VPs regarding approval of projects. Ministerial direction is also sought for some ICF amendments considered to be of a substantive nature, as described in the Agency’s policy on “Amendments to Contribution Agreements.”

In the case of the BDP, project approval authority is delegated to various levels depending on the regional office and type of project, and is also established as per the Agency’s financial signing authority delegation instrument. The BDP levels of authorization are limited by the total amount of the ACOA grant and contribution to a particular project. Within each region, authority for approval levels under $500,000 vary; in general, however, most decisions are made within the delegated authority of the regional VPs. ACOA’s president or VP of policy and programs can approve contributions up to $1,000,000, while the minister approves contributions up to $10,000,000.

Within ACOA, there are functional leads on priorities such as tourism, Indigenous communities, OLMCs and business skills development. These functional leads play a support and coordination role for CM and CI programming that contributes to these priorities.

2.5 Expenditure Profile

2.5.1 Proportion of Overall Agency Expenditures

Between 2008-2009 and 2011-2012, CM and CI programming represented 28 per cent of Agency expenditures. Table 3 provides an overview of ACOA’s expenditures by program over the evaluation period.

Table 3: Expenditures ($M) by Program, 2008-2009 to 2011-2012

Program	Expenditures
Program	2008-2009	2009-2010	2010-2011	2011-2012	Total
Enterprise Development	215.0	192.2	191.0	172.9	771.1
Community Development	113.1	155.7	161.9	103.8	534.5
Community Mobilization and Community Investment Sub-Programs*	76.6	126.9	134.7	73.7	411.9
Policy, Advocacy and Coordination	13.3	13.0	13.7	13.7	53.7
Internal Services	0	42.9	41.8	40.0	124.7
Total Expenditures	341.4	403.8	408.4	330.4	1,484.0

Source: Departmental Performance Reports.

*Data for CM and CI came from the GX financial systems.

Note that for 2008-2009, expenditures for internal services in support of program functions were reported within the program areas, rather than separately. Removing the expenditures dedicated to internal services in support of CM and CI programming for 2008-2009 produces an adjusted total of $76.2 million, for an adjusted total of $411.5 million over the four year period.

2.5.2 CM and CI Expenditures

As can be seen in Table 4, grant and contribution (G&C) expenditures represent the majority of program spending (92 per cent), followed by salaries (6 per cent) and operating expenditures (2 per cent). G&C expenditures increased in 2009-2010 and 2010-2011 due to the delivery of initiatives associated with the EAP . The slight increase in salaries over this period was also related to the EAP .

Table 4: Annual Expenditures ($M) for CM and CI, 2008-2009 to 2011- 2012

Year	G&Cs	Salaries	Operations	Total
2008-2009	68.5	5.6	1.8	75.9
2009-2010	118.2	6.9	2.0	127.1
2010-2011	127.2	6.2	1.5	134.9
2011-2012	66.4	6.0	1.2	73.6
Total	380.3	24.7	6.5	411.5

Source: ACOA GX Financial Database (Extracted June 25, 2012).

Table 5 details annual G&C expenditures by transfer payment program. Over the evaluation period, BDP expenditures increased by 35 per cent and ICF expenditures decreased by 32 per cent. The majority of CM projects are supported through the BDP; CI projects were supported using funds from both the ICF and the BDP. EDI expenditures in support of the Roadmap for Canada’s Linguistic Duality are included in Table 5 and were delivered using ICF and BDP funds. As mentioned above, CM and CI staff also delivered the bulk of CAF and RInC funding, mostly in 2009-10 and 2010-11.

Table 5: Annual G&C Expenditures ($M) for CM and CI, 2008-2009 to 2011- 2012

Year	Expenditures by Transfer Payment Program				Total
Year	BDP	ICF	RInC	CAF	Total
2008-2009	19.7	48.8	0	0	68.5
2009-2010	27.1	45.6	11.3	34.2	118.2
2010-2011	29.8	31.7	19.4	46.3	127.2
2011-2012	30.2	33.1	3.1	0	66.4
Total	106.8	159.2	33.8	80.5	380.3

Source: ACOA GX Financial System (Extracted June 25, 2012).

2.6 Project Profile

To create a meaningful representation of funded projects than was possible through ACOA’s project database (QAccess), the evaluation team undertook an analysis of the 1,087 CM and CI projects supported over the evaluation period, excluding 230 RInC projects ($34.2 million), 167 CAF projects ($82.9 million), 34 EDI projects ($5.19 million) and the NB Francophone Immigration Initiative ($10 million) approved over the evaluation period. The analysis resulted in the creation of two broad categories – planning projects and investment projects. Table 6 shows the number of projects and spending amounts approved for both planning and investment projects over the evaluation period.

Table 6: Planning and Investment Projects Approved, 2008-2009 to 2011- 2012

Type of Community Economic Development Project	Number of Projects	ACOA Approved ($M)
Planning	338	42.6
Investment	749	184.3
Total	1,087	226.9

Planning projects included planning for the economic development of a geographic area, industry sector or specific initiatives. Investment projects included funding for: construction or equipment purchases in support of economic development for communities and sectors; marketing initiatives; events; business or technical skills development; and research in support of sector development, among others. The types of projects supported within each broad category will be presented in more detail in Section 4.

3. Findings: Relevance

Overall, the evaluation found that the CM and CI sub-programs are relevant activities for ACOA and the federal government. There is a continued need for planning and investment programming as challenges to CED exist to the same or a greater degree than reported in the previous evaluation. In response to changing CED needs, the Agency has made adjustments to programming, including an increased emphasis on investments that generate greater economic impacts and the cessation of operational support to REDOs across Atlantic Canada. CM and CI activities align with ACOA’s strategic outcome and the ACOA Act, and are consistent with the Agency’s roles and responsibilities and broader federal priorities. In general, the Agency’s programming complements rather than duplicates that of other organizations.

In generating these findings, the evaluation team assessed the relevance of CM and CI programming by examining: (1) the continued need for the programming; (2) the programming’s alignment with the Government of Canada’s priorities and with the Agency’s priorities; and (3) federal roles and responsibilities.

3.1 Continued Need for Programming

Judgment Criteria	Key Finding
The programming-related needs are still present to at least the same degree as they were five years ago.	Most community economic development challenges exist to at least the same degree or are greater than what was reported in the previous evaluation. There was a substantial increase in perceived challenges of an aging population and outmigration since the last evaluation.
Community planning and investments respond to emerging needs.	ACOA’s CM and CI programming responds to emerging needs. Contributing factors include the Agency’s decentralized delivery model; the support provided from the PAC function and ED programming; and stakeholder relationships.
ACOA is aware of changing economic development needs and adjustments to programming are made to meet those needs.	ACOA is aware of changing CED needs and has made some adjustments to programming over the evaluation period. These include the proactive identification or development of projects with greater potential economic impacts and the decision to discontinue operational funding to REDOs.
The roles and responsibilities of other service providers do not duplicate those of ACOA (extent of duplication, overlap or complementarity).	ACOA’s CM and CI programming is unique and complementary to CED programming offered by other organizations. The CM and CI programming’s distinctive traits include its Atlantic-wide scope,; PAC contributions, internal capacity for proposal development and assessment, programming flexibility, and focus on CED goals.

3.1.1 Programming-Related Needs

Over the past five years, economic development in rural areas has continued to be challenged by the decline of traditional industries, population shifts, infrastructure issues and other factors described below. These issues emphasize the need to support initiatives that increase the competitiveness of Atlantic Canadian rural communities and businesses. Such initiatives include strategic economic development planning and infrastructure required to attract and retain labour; transportation and other types of infrastructure that support economic activity and specific sectors; and marketing and skills development.

Global Economy and Decline in Traditional Industries

The global economic downturn negatively impacted traditional industries and employment in Atlantic Canada. By late 2008, Atlantic Canada posted reduced gross domestic product growth. While the Atlantic Canadian economy began to recover in 2010-2011, growth was less stable in 2012 across Canada due, in part, to economic difficulties abroad that affected the global economy. [12] VII]

Increasing global competition from lower cost and/or higher-value production in emerging markets, combined with the economic downturn, negatively affected traditional industry sectors such as fishing, forestry and agriculture, which have long been important in Atlantic Canada, particularly in rural areas. [13],[14] Particular challenges include structural adjustments in the fishing sector, mill closures and job losses in forestry, and the loss of market shares in the agricultural sector. [15] For instance, over a quarter (26 per cent) of clients surveyed for this evaluation stated that industry/company closures are a major challenge to CED, compared to 13 per cent in the previous evaluation. [16]

Company closures related to these challenges contributed to an increase in unemployment following 2008. The Atlantic provinces continue to experience unemployment above pre-recession levels, with the exception of Newfoundland and Labrador, where in 2012, the number of people without a job was lower than in 2008, attributable to growth in the oil and gas and mining sectors. [17] Over half of surveyed clients (52 per cent) reported lack of employment opportunities as important barriers to CED, a significant increase over the previous evaluation (21 per cent). [18]

Demographic Challenges, Labour Shortages and Skills

Atlantic Canada’s aging population, out-migration and related lack of skilled workers and volunteers have continued to challenge economic development over the evaluation period. The Atlantic Canadian population is older than that in other Canadian regions due to low birth rates, modest immigration levels and high out-migration rates. [19] Nearly half (48 per cent) of surveyed clients cited an aging population as a key barrier to economic development, compared to 12 per cent in the previous evaluation, while 45 per cent cited out-migration as a key challenge, compared to 15 per cent in the previous evaluation.

These demographic issues have contributed to Atlantic Canada’s growing labour shortages. [20] In a 2011 survey of 600 Atlantic Canadian businesses, 59% reported having difficulty recruiting qualified workers. [21] This issue was more pronounced in rural areas. According to a 2012 report by the Atlantic Provinces Economic Council, Atlantic labour force growth is primarily urban, and the disparity between National and Atlantic participation rates is due to declines in rural regions. [22] The report further states that a specialized skills and expertise are critical to the production of innovative and value-added goods and services, which are necessary for improved productivity and expansion into new markets.

Business skills within SMEs are also critical to the economic growth of Atlantic Canada. Self-employment in Atlantic Canada lags behind other Canadian regions. [23] A 2007 research report by the Institute for Competitiveness and Prosperity suggested that strengthening management skills will lead to greater innovation, productivity, international competitiveness and, ultimately, the prosperity of Canadians. [24] This was echoed by key informants and in case studies completed through this evaluation, where lack of business skills was strongly linked to projects not achieving expected outcomes.

Much like businesses, non-profit organizations need skilled employees and volunteers to manage effectively. According to a 2006 study of non-profit organizations in Atlantic Canada, over half of all organizations in the region reported financial, human and organizational capacity as barriers. [25] The challenge of a skilled labour shortage was also raised by key informants consulted through this evaluation.

Infrastructure Issues

Basic infrastructure such as transportation access, water/sewage, broadband and telecommunication, and community/recreational facilities are considered essential to the development of local and regional economies. [26] While the national average age of public infrastructures decreased from 17.5 years in 2000 to 16.3 years in 2007, three Atlantic provinces were among those with the oldest public infrastructures in Canada: Nova Scotia (18 years), Newfoundland (17. 3) and New Brunswick (16.9). [27] Close to half of clients (47 per cent) surveyed indicated infrastructure issues as posing challenges to community economic development, representing an increase from the previous evaluation (13 per cent).

Access to Funding

Governments are the single most important source of funding for non-profit and voluntary organizations, accounting for 45 per cent of all funding in the Atlantic region and 49 per cent nationally. [28] As the Canadian economy began recovering from the recession in 2010 -2011, the Government of Canada was reducing spending and provincial governments were also in a period of fiscal restraint. [29] Since municipalities, chambers of commerce, Indigenous groups, social enterprises and other industrial/sector associations often seek provincial and federal funding for economic development initiatives, there were greater challenges in leveraging resources over the evaluation period. Of clients surveyed, 58 per cent identified access to funding as a barrier, compared to 19 per cent in the previous evaluation. Key informants noted that fewer private-sector funds are available to support CED projects in the Atlantic region compared to other Canadian regions due to a reduced presence of private-sector and decision-making centres of larger national or multinational firms.

Francophone and Indigenous Communities

As in other regions of Canada, francophone and Indigenous communities in Atlantic Canada experience specific challenges to economic development. According to Statistics Canada data, challenges faced by Canadian OLMCs include greater population decline, out-migration, an aging population, unemployment and low self-employment, and greater employment in traditional industries, which are under pressure. [30] A 2012 evaluation of the ACOA-delivered New Brunswick francophone immigration initiative determined that programming to encourage francophone immigration was necessary to help the Province meet demographic challenges and contribute to its economic, cultural and social future. The recent EDI evaluation concluded that there is an important continuing rationale for programming in support of economic development initiatives for OLMCs. Challenges relating to the economic progress of Indigenous communities include health issues, lower levels of educational attainment, greater unemployment and lower income, reflecting an ongoing need for targeted economic development initiatives. [31]

3.1.2 Awareness of and Responsiveness to Changing Needs

ACOA remains aware of changing economic development needs and opportunities through its decentralized delivery model (i.e. regional and district offices), close partnerships with a variety of government and non-governmental stakeholders, and the knowledge and resources contributed by PAC and ED . In response to changing needs, the Agency has made some adjustments to programming, including the cessation of REDO operational funding, the implementation of the Government’s EAP as well as funding projects with greater potential economic impacts.

Key informants indicate that the Agency’s flexible and decentralized delivery model allows for a better understanding of regional and local issues through direct contact with clients, communities and other key stakeholders such as provincial and local governments. According to key informants, relationships with and contributions from ACOA PAC and ED programming enhance the Agency’s understanding of, and response to, CED challenges. Examples include teamwork between the units aimed at stimulating economic activity in response to a pulp and paper mill closure in Newfoundland and Labrador or maximizing opportunities for a region’s businesses to participate in a global supply chain in New Brunswick.

The Government’s EAP initiatives, RInC and CAF, were delivered by ACOA in Atlantic Canada in response to global economic decline. The programming was mainly targeted at declining infrastructure and other investments to support key industries. As previously stated, in deciding to discontinue operational funding to REDOs, ACOA considered the changing demographics of rural areas and the need to provide greater focus to its other investments so as to maximize economic growth and job creation in response to the global recession and other challenges to economic development. Examples of such initiatives include integrating businesses affected by the decline of traditional industries into the global value chain; technological innovations needed to train the labour force for growing industries; and the development of infrastructure that enables economic development to take place, such as infrastructure related to transportation, electrical upgrades and broadband access..

3.1.3 Extent of Overlap, Duplication and Complementarity with Other Programming

ACOA’s CM and CI programming is unique and complementary to programming offered by other governments and organizations. While other organizations provide some support for CED, a number of factors contribute to the uniqueness and importance of ACOA’s CM and CI programming in the region, particularly the amount and type of funding support, its Atlantic-wide scope of intervention, its targeted focus on CED, its decentralized delivery model and the coordination with PAC and ED programming.

Other than ACOA, provincial governments are the most regular co-funders of CM and CI projects. Over the evaluation period, the Provinces co-funded 65 per cent of projects, with their contributions representing 26 per cent of total costs. Provincial involvement was similar for both planning and investment projects. Each Atlantic Province has a department that commonly co-funds community infrastructure projects, and all four Provinces have other departments that sometimes contribute funding to sector-specific initiatives. According to key informants, provincial departments often have smaller budgets and less capacity for project assessment and development. Client organizations and other external funding agents confirmed the importance of ACOA’s role in project assessment and development, which often influences the decisions of provincial governments and others to invest in CED initiatives.

ACOA’s programming distinguishes itself from that of other federal government departments because of its specific focus on CED. ACOA’s decentralized delivery model positions it well for delivery of federal economic development initiatives such as the EAP . The Agency’s regional presence allows for better relationships with regional stakeholders and clients and supports consideration of contextual factors. Key informants noted that ACOA plays a key role in fostering collaboration and coordination, as well as the development of strategies and investment projects among federal partners.

ACOA’s unique focus on Atlantic Canada allows it to respond to specific regional needs, opportunities and contextual factors, and to take advantage of synergies between the Provinces. Furthermore, a unique characteristic of ACOA’s CM and CI programming is the ability to draw from PAC research capacity and ED knowledge relating to commercial activity to inform planning, project development and decision making. For example, in Nova Scotia, CD and ED account managers collaborated on an aquaculture association initiative to promote industry best practices and increase community knowledge of aquaculture. The aim was to dispel myths and demonstrate the diversity of aquaculture operations, leading to greater awareness and capacity to pursue commercial opportunities. In Newfoundland, CD and ED programming jointly supported efforts to integrate regional businesses into the supply chain for a major oil and gas project. CI programming contributed to enhancing awareness of opportunities, facilitated networking with major purchasers, and enhanced business skills related to supplying major projects. ED programming supports were then used to work directly with firms to address identified needs (e.g. ISO certification).

As previously mentioned, over the evaluation period, ACOA discontinued operational funding to REDOs. Following the withdrawal of ACOA funding, the landscape of key players in regional economic development planning and mobilization changed and continues to shift. Key informants cautioned that this period of transition will require ACOA to adjust to changes in key stakeholders, and will require the monitoring of the potential risk of capacity gaps.

Organizations that provide advice and training to support entrepreneurs, such as the Community Business Development Corporations, University Business Development Centres in Nova Scotia, and other provincially funded programming, complement some CI initiatives aimed at building the business skills and knowledge of entrepreneurs.

For more detail on other organizations that support similar programming, see Appendix E.

3.2 Alignment with ACOA and Government-wide Priorities and Strategies

Judgment Criteria	Key Findings
There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome.	ACOA’s CM and CI programming aligns with the Agency’s strategic outcome and the priorities of the Government.

As indicated in ACOA’s PAA, communities are critical for prosperity. They have varying capacities, challenges and strengths and their economic development requires a flexible, holistic approach. [32] As described in the PAA, by mobilizing key stakeholders for the planning and implementation of economic development initiatives, CM and CI sub-programs support the broader goal of the CD program, which is to foster “dynamic and sustainable communities for Atlantic Canada,” in support of ACOA’s strategic outcome of “A competitive Atlantic Canadian economy”. For example, the evaluation’s project analysis shows that ACOA has supported planning activities related to the economic development of geographic regions and sectors of interest. ACOA investments included initiatives that support the development of growing and traditional industry sectors and community infrastructure, such as high- quality recreational or cultural infrastructure, that attract and retain skilled labour.

According to key informants and project analysis, CM and CI programming has also supported Agency priorities through projects aimed at economic development in francophone and Indigenous communities, projects such as the Celtic Colours Festival and Joggins Fossil Cliffs that support ACOA’s Strategy for tourism growth, and projects that support the development of business skills in Atlantic communities, such as the Black Business Initiative in Nova Scotia, training on innovation and product development in NB and entrepreneurship and business skills promotion in PEI.

With respect to supporting Government of Canada priorities, as previously discussed, ACOA delivered the EAP initiatives, which were a high priority during the evaluation period. There is also explicit and logical alignment between the planning and implementation of economic development initiatives and one of the high-level expected outcomes identified in the Government of Canada’s whole-of- government framework: “strong economic growth,” which is focused on increasing economic growth and development in all regions and all sectors of the economy. [33] ACOA’s programming supports other Government priorities as evidenced by links to speeches from the throne and federal budgets over the evaluation period. For example, ACOA supported the development of traditional industries[34] through investments in economic development infrastructure and training relating to the mining, marine and agriculture sectors. ACOA also supported infrastructure construction[35] related to recreation, transportation and business (e.g. electrical upgrades). Finally, ACOA supported economic opportunities for Indigenous Canadians[36] and skills development and training,[37] as described above, and francophone identity and linguistic duality[38] through the implementation of the EDI.

3.3 Alignment with Federal Roles and Responsibilities

Judgment Criteria	Key Findings
ACOA’s roles and responsibilities in this area of programming are aligned with the ACOA Act.	CM and CI programming aligns with the ACOA Act.
Similar federal organizations in other jurisdictions administer such programming.	ACOA’s CM and CI programming is similar to what is offered through other RDAs.

The ACOA Act gives the Agency the authority to “plan, direct, manage and implement programs and projects intended to contribute directly or indirectly to the economic prosperity of the Atlantic region.”[39] By planning and investing in infrastructure and other initiatives that support economic development in rural Atlantic Canada, CM planning and CI investment activities contribute to ACOA’s mandate to “increase opportunity for economic development in Atlantic Canada and, more particularly, enhance the growth of earned incomes and employment opportunities in that region.”[40]

A comparison of ACOA CM and CI programming with that offered by other RDAs also supports CM and CI alignment with federal government roles and responsibilities. As seen in Table 7, all five RDAs deliver services and programming tailored to regional needs and opportunities, with the aim of helping communities adapt to economic change and strengthen their economies. Several RDAs provide non-repayable grants and contributions to not-for- profit organizations through application-based processes to fund activities for strategic/sectoral planning, skills development, community infrastructure and equipment purchasing. In all cases, RDAs provide programming dedicated to official language minorities and Indigenous communities. Other RDAs also deliver community mobilization and planning activities through the national Community Futures Program as well as the aforementioned EDI and EAP initiatives. Most deliver Infrastructure Canada programming.

Table 7: Programming Similar to CM and CI Delivered by RDAs Across Canada

RDA	Type of Programming
RDA	Sectoral/regional planning	Marketing	Training /skills development	Community/ recreational infrastructure	Industry/sector infrastructure	Equipment	Tourism
ACOA	X	X	X	X	X	X	X
CED-Q	X			X	X		X
WD	X	X	X		X	X
FedDev	X		X	X
FedNor	X	X	X		X	X	X
CanNor	X	X	X	X	X	X	X

RDA	Horizontal Programming		EDI	Indigenous
RDA	Community Futures Program	Infrastructure Canada	Support official language minority	Support Indigenous communities
ACOA	X	X	X	X
CED-Q	X	X	X	X
WD	X	X	X	X
FedDev	X	X	X	X
FedNor	X		X	X
CanNor	X		X	X

4. Findings: Performance – Effectiveness

Overall, the evaluation found that the CM and CI programming is incremental or critical to the implementation of projects and the achievement of outcomes. Analysis of project data demonstrated that the construction of community, recreational and/or cultural infrastructure is a large component of CI programming, as is support to tourism. CI objectives with respect to these areas are not clearly articulated in program documentation.

The majority of planning and investment projects surveyed achieved immediate outcomes and many investment projects achieved intermediate outcomes. The survey provided some indication of longer- term outcomes achieved, and case studies highlighted the importance of planning to achieving desired outcomes. The evaluation provides evidence that ACOA investments contributed to strengthening the capacity and asset base of communities for economic development, but cannot speak to related impacts on the economic viability of communities. Some barriers to achieving results exist and could be better mitigated, particularly through improved project viability assessment processes. Limitations with performance measurement information present challenges for strategic and efficient program management.

The evaluation team assessed the effectiveness of the CM and CI sub-programs by examining: (1) incrementality; (2) evidence of achievement of expected results; (3) unexpected outcomes; (4) overall achievement of CED impacts; (5) barriers to achieving results, (6) lessons learned and best practices; and (7) evidence that performance measurement is adequate and effective. As previously stated, the evaluation of effectiveness (outcome achievement) does not include programming delivered under the EAP and the Roadmap for Canada’s Linguistic Duality.

4.1 Incrementality

Judgment Criteria	Incrementality Key Findings
The impact of the absence of the programming is similar to or greater than that observed in the previous evaluation, as reflected by: ACOA clients’ opinion of whether the project/initiative would have proceeded without funding; and ACOA’s influence on the involvement of funding partners.	CM and CI programming was incremental to the implementation of projects, to obtaining investments from other funding partners and to achieving results. According to survey respondents, the majority of projects would likely not have proceeded without ACOA’s funding. Of those projects that could have proceeded, almost all would have experienced negative impacts, including smaller scope, time delays, lower quality, and difficulty securing other funding. ACOA provides a large percentage of total funding to planning and investment projects and influences the involvement of other funding partners.

Judgment Criteria

Incrementality Key Findings

The impact of the absence of the programming is similar to or greater than that observed in the previous evaluation, as reflected by:

ACOA clients’ opinion of whether the project/initiative would have proceeded without funding; and
ACOA’s influence on the involvement of funding partners.

CM and CI programming was incremental to the implementation of projects, to obtaining investments from other funding partners and to achieving results.
According to survey respondents, the majority of projects would likely not have proceeded without ACOA’s funding. Of those projects that could have proceeded, almost all would have experienced negative impacts, including smaller scope, time delays, lower quality, and difficulty securing other funding.
ACOA provides a large percentage of total funding to planning and investment projects and influences the involvement of other funding partners.

To gauge the degree to which results being reported are attributable to ACOA’s investments, clients were asked what would have occurred in the absence of the funding received. As shown in Figure 1, almost two thirds of the clients reported that their projects would likely not have proceeded at all (full incrementality). A third of clients reported that their projects would likely have experienced major negative impacts, including problems securing other funding (31 per cent), reduced scope (29 per cent), lower quality (23 per cent) and delays in initiating or completing the project (15 per cent). Only 3 per cent of clients surveyed reported that their projects would have proceeded with only minor negative impacts in the absence of ACOA support.

Figure 1: Incrementality of CM and CI Programming

(D)

Full incrementality is slightly higher for planning (68 per cent) than for investment projects (63 per cent). Of investment projects, full incrementality is highest for projects related to construction (68 per cent) and equipment (78 per cent), and lower for events (28 per cent). The degree of full incrementality found in this study is slightly lower than, though still consistent with, what was found in the previous evaluation, in which 69 per cent of investment projects and 73 per cent of planning projects would likely not have proceeded without ACOA’s investment. [VIII]

Another indicator of incrementality is that ACOA influences other partners to support planning and investment projects. The majority (70 per cent) of clients indicated that other funding was secured as a result of ACOA’s decision to provide support. Similarly, 76 per cent indicated that it would have been unlikely or very unlikely that they could have obtained other funding without ACOA’s support. On average, clients reported that 42 per cent of total project funding was secured from external sources as a result of ACOA’s support. Key informants report that ACOA’s investments have an important influence on leveraging other project funding, particularly from provincial governments but also from municipal governments and other federal departments.

4.2 Achievement of Expected Results

Overall, the evaluation has found that the majority of both planning and investment projects achieved the immediate results expected of CM and CI programming, such as enhanced collaborations and partnerships for economic development. Planning resulted in capacity building and developed plans while investment projects resulted in the creation of infrastructure for economic development and increased capacity for responding to economic opportunities and crises. While it was not possible to comment on the extent of achievement of the intermediate results of planning projects, there is evidence that investment projects contributed to increased tourism visitation, the development of new products, and increased sales and revenues for businesses.

4.2.1 Planning Projects

In general terms, planning projects were expected to mobilize partners, build capacity for planning and develop plans for CED. Planning projects undertaken over the evaluation period achieved these immediate outcomes by increasing CED skills and leadership, influencing decision making and leading to concrete actions for economic development.

The client survey response rate for planning projects was good (77 per cent), however the exclusion of REDOs from the survey and the consideration of completed projects only resulted in a small number of responses on outcome achievement (32). Though survey findings on the outcomes of planning projects are not statistically representative of all ACOA planning projects, they provide some indication of outcomes achieved,[IX] and case studies provided valuable information on the importance of planning for the achievement of these longer-term outcomes.

Types of Planning Projects

There were four main types of planning projects over the evaluation period: 1) support for REDO operations, including core planning activities; 2) other regional economic development planning; 3) industry sector development planning; and 4) studies or plans related to specific projects or initiatives. Planning project proponents included REDOs, industry associations, indigenous organizations, municipalities and other non-governmental organizations. Table 8 outlines the number of approved projects and amounts of approved funding by project type.

Over half of ACOA approved funding for planning projects (54 per cent) supported REDOs’ operational activities, including their core planning activities. The second highest amount of funding approved was related to planning for the development of industry sectors (17 per cent) such as tourism, renewable/alternative energy, fisheries and agriculture/agri- foods. The greatest number of planning projects (15 per cent of the funding) consisted of project-specific plans or planning/studies that support decision making related to a more specific initiative. These included feasibility studies, needs assessments and engineering or building plans. Finally, 14 per cent of funding supported regional economic development plans that were not part of REDOs’ core operational planning activities. These generally focused on economic development for Indigenous, francophone and other rural communities.

Table 8: Types of Planning Projects and Funding Approved, 2008-2009 to 2011- 2012

Planning Project Type and Description	ACOA Projects Approved		ACOA $ Approved
Planning Project Type and Description	Number	%	$M	%
REDO operational Operational funding for REDOs	89	26	23.0 [X]	54
Industry-related planning Planning for the development of a specific industry such as tourism, oil and gas, or mining	69	21	7.4	17
Project-specific planning Planning for particular initiatives, includes feasibility studies, strategic plans, construction plans and marketing strategies	116	34	6.2	15
Regional economic development planning Planning for community economic development that is not included in REDO operational funding	64	19	6.0	14
Total	338	100	42.6	100

Source: QAccess and detailed project coding data.

REDOs were involved in all types of planning projects in addition to core planning activities undertaken as part of their operations. They acted as proponents for 25 per cent of the other three types of planning projects. REDOs conducted 22 projects (14 per cent of the funding) related to other regional economic development planning, 13 projects (7 per cent of the funding) for industry-related planning, and 26 projects (27 per cent of the funding) for project-specific planning activities. Key informants also suggested that REDOs were collaborators on many types of projects.

Immediate Outcomes of Planning Projects

Judgment Criteria	Key Findings
The contribution of planning activities to immediate outcomes is similar to or greater than that observed in the previous evaluation as reflected by: at least 50 per cent of projects will result in increased community or organization capacity for strategic planning relating to economic development; at least 50 per cent of projects will result in increased leadership for strategic planning relating to economic development; at least 60 per cent of projects will result in improved collaboration relating to economic development within the community or other organizations (stakeholders); 90 per cent of strategic plans have been finalized plans meet “quality” criteria that is more likely to lead to economic development outcomes; and/or strategic plans identify opportunities that can lead to economic impacts such as employment/business creation, sustainability or growth.	There was evidence of the achievement of immediate expected outcomes of planning projects. Of 32 completed projects surveyed: 88 per cent (28 projects) reported improved ability to conduct economic development planning; 81 per cent (26 projects) reported increased business skills; 91 per cent (29 projects) reported improved leadership for strategic planning; and 94 per cent (30 projects) reported improved collaboration for economic development. The average number of organizations involved in a project was19, though the majority of projects involved three to five other organizations. According to ACOA’s departmental performance reports (DPRs) over the evaluation period, almost all REDOs (92 per cent) finalized strategic plans. Due to the small number of planning projects that were part of the survey, the evaluation could not conclude on the quality of plans or the proportion of plans that identify opportunities that can lead to economic impacts.

Judgment Criteria

Key Findings

The contribution of planning activities to immediate outcomes is similar to or greater than that observed in the previous evaluation as reflected by:

at least 50 per cent of projects will result in increased community or organization capacity for strategic planning relating to economic development;
at least 50 per cent of projects will result in increased leadership for strategic planning relating to economic development;
at least 60 per cent of projects will result in improved collaboration relating to economic development within the community or other organizations (stakeholders);
90 per cent of strategic plans have been finalized
plans meet “quality” criteria that is more likely to lead to economic development outcomes; and/or
strategic plans identify opportunities that can lead to economic impacts such as employment/business creation, sustainability or growth.

There was evidence of the achievement of immediate expected outcomes of planning projects. Of 32 completed projects surveyed:

88 per cent (28 projects) reported improved ability to conduct economic development planning;
81 per cent (26 projects) reported increased business skills;
91 per cent (29 projects) reported improved leadership for strategic planning; and
94 per cent (30 projects) reported improved collaboration for economic development. The average number of organizations involved in a project was19, though the majority of projects involved three to five other organizations.

According to ACOA’s departmental performance reports (DPRs) over the evaluation period, almost all REDOs (92 per cent) finalized strategic plans.

Due to the small number of planning projects that were part of the survey, the evaluation could not conclude on the quality of plans or the proportion of plans that identify opportunities that can lead to economic impacts.

The 32 clients with completed plans (representing 64 per cent of invited survey participants) reported the following immediate outcomes either to some extent or to a great extent:[XI]

improved leadership for strategic planning (91 per cent, or 29 projects);
improved ability to identify economic opportunities (88 per cent, or 28 projects);
improved ability to conduct economic development planning (81 per cent, or 26 projects); and
increased business skills (81 per cent, or 26 projects).

Less than 1 per cent of respondents reported that the above outcomes were “not applicable.”

According to survey results, 30 out of 32 completed planning projects (94 per cent) resulted in improved collaboration for economic development. The average number of other organizations involved in projects was 19, though the majority of projects involved three to five other organizations. According to client survey responses, provincial governments (93 per cent) were the most frequent collaborator, followed by municipal governments (80 per cent), industry (68 per cent), other federal governments (61 per cent), non-governmental organizations (59 per cent), academic organizations (58 per cent) and Indigenous governments (54 per cent). All case studies highlighted the importance of collaboration as both a facilitator to project success as well as a key outcome.

Case Study – Planning: Enhanced Collaboration and the Development of Plans

ACOA supported the development of a lobster value and marketing strategy in response to challenges faced by the industry in Atlantic Canada.

Engagement and consensus-building activities resulted in increased collaboration among stakeholders in this complex and fragmented industry. The project led to a better understanding of the industry and provided needed direction.

A long-term value and marketing strategy supporting the future competitiveness of the sector was finalized and activities are being implemented.

As reported in the Agency’s DPRs, it met targets for the expected number of collaborators involved in REDO operational planning projects 92 per cent of the time over the four fiscal years covered by the evaluation. [41] The total number of collaborators ranged from 333 in 2011-2012 to 527 in 2008-2009, an average of 6 to 10 collaborators per REDO.

ACOA’s investments in planning projects resulted in the development of different types of plans. As previously stated, analysis of project data reveals that activities focused on the development of regional economic development plans, business plans, marketing strategies, construction plans, event plans and others.

ACOA met targets set for the presence of regionally based economic development plans, as reported by REDO operational projects. The number of plans ranged from 48 to 60 per year over the four years, meaning that an average of 92 per cent of REDOs developed regional economic development plans each year. The evaluation did not directly assess the quality or implementation of these plans.

Of the 32 survey participants that reported completed planning projects, 29 (91 per cent) reported that a plan was finalized.

Intermediate Outcomes of Planning Projects

Judgment Criteria	Key Findings
Plans contain initiatives that have been implemented and that have increased capacity for economic development, as reflected by: jobs created or maintained; reduced out-migration; the creation or survival of businesses; increased sales and revenues for businesses in the region; the attraction of investment; increased exports; the development of new markets; diversified exports; product development; the development of business skills; and tourism visitation influenced by planning activities.	While the numbers are too small to generalize, survey responses for completed planning projects indicate that the plan helped proponents make decisions regarding new initiatives, and increased their ability to respond to opportunities. Plans also resulted in concrete actions, the creation of infrastructure, the acquisition of new financial resources for initiatives, the development of new projects and the creation/ maintenance of jobs. Increased tourism and increased sales and revenues for business were also reported. Outcomes related to exports and reduced out-migration were not commonly reported and may not be highly applicable to planning projects.

Judgment Criteria

Key Findings

Plans contain initiatives that have been implemented and that have increased capacity for economic development, as reflected by:

jobs created or maintained;
reduced out-migration;
the creation or survival of businesses;
increased sales and revenues for businesses in the region;
the attraction of investment;
increased exports;
the development of new markets;
diversified exports;
product development;
the development of business skills; and
tourism visitation influenced by planning activities.

While the numbers are too small to generalize, survey responses for completed planning projects indicate that the plan helped proponents make decisions regarding new initiatives, and increased their ability to respond to opportunities. Plans also resulted in concrete actions, the creation of infrastructure, the acquisition of new financial resources for initiatives, the development of new projects and the creation/ maintenance of jobs. Increased tourism and increased sales and revenues for business were also reported. Outcomes related to exports and reduced out-migration were not commonly reported and may not be highly applicable to planning projects.

Survey respondents that had completed their planning projects (32) reported that as a result of their planning activity:[XII]

they were able to make decisions regarding future initiatives (100 per cent, or 29 projects);
concrete actions took place as a result (88 per cent, or 28 projects);
initiatives were implemented to respond to an economic opportunity or crisis (86 per cent, or 25 projects);
economic development infrastructure was created ( or 16 projects); and
their ability to respond to future opportunities or crises improved (86 per cent, or 25 projects).

Survey findings also provided some indication of the types of longer-term outcomes that may be more commonly achieved:[XIII]

new financial resources for initiatives were obtained (63 per cent, or 20 projects);
new products were developed (59 per cent, or 19 projects);
jobs were created or maintained (56 per cent, or 18 projects);
tourism visitation increased (56 per cent, or 18 projects);
sales and revenues for businesses increased (53 per cent, or 17 projects);
new investments were obtained (47 per cent, or 15 projects); and
businesses were created or maintained (41 per cent, or 13 projects);

Over half of the 32 organizations with completed projects reported that intermediate outcomes related to exports and reducing out-migration were not applicable to their projects. These survey results combined with information collected through case studies suggest that outcomes related to exports and reducing out-migration may not be directly relevant for planning activities.

4.2.2 Investment Projects

Types of Investment Projects

Table 9 presents the types of investment projects that ACOA funded over the evaluation period, based on detailed analysis of project data.

Investment Project Type and Description	Projects		Approved Funding
Investment Project Type and Description	Number	%	$M	%
Construction/Infrastructure Construction of community buildings, recreational facilities, cultural sites or industry-related infrastructure	291	39	112.3	61
Marketing Promotion of tourism regions/events/attractions and other industry sectors	117	16	26.0	14
Equipment Purchase or lease of equipment for recreational/cultural events and facilities, or industry sector development, training and research	73	10	14.3	8
Events Festivals, conferences and other events relating to recreation/culture, industry sectors or businesses	166	22	13.2	7
Skills Development Training or other initiatives to increase business or technical knowledge and skills	76	10	10.4	6
Research Centre Operation Research centres related to specific industry sectors such as mining and forestry	5	1	4.6	2
Tourism Operation Operation of historic sites, performing arts facilities, museums or interpretative centres	11	1	1.6	1
Immigration/recruitment Investments to attract immigrants and recruit skilled labour	9	1	1.8	1
Total	748	100	184.2	100

Source: QAccess and project coding analysis.

The largest proportion of investment funding was for the construction/renovation of infrastructure (61 per cent). Marketing initiatives represented 14 per cent of investment funding. A large majority of funding for marketing projects was related to tourism, including a $9 million Atlantic tourism marketing initiative. Other types of investment projects included acquiring equipment (8 per cent), implementing events (7 per cent) and developing business or technical skills (6 per cent). Less frequent types of investment projects supported the operation of research centres (2 per cent), the operation of tourism attractions (1 per cent) and immigration or recruitment of skilled workers (1 per cent). See Appendix F for a more detailed analysis of project sub-types and Appendix G for main project types by ACOA region.

Thirty-three percent of investment funding ($61.1 million), representing over half of the funding approved for construction/infrastructure projects, supported community, recreational and cultural infrastructure:

21 per cent of funding (34 projects; $23.3 million) was for community buildings, including multi-purpose centres;
19 per cent of funding (70 projects; $21.0 million) was for cultural/heritage initiatives such as museums and interpretative centres and performing arts facilities; and
15 per cent of funding (76 projects; $16.8 million) was related to recreation infrastructure such as marinas, sports facilities and parks.

These projects, their expected outcomes and their role in achieving CI objectives are not well articulated in CI programming documents.

Project analysis for the years 2009-2010 to 2011-2012[XIV] also revealed that a large proportion of investment funding was linked to tourism. Many projects aligned directly with ACOA’s Growth Strategy for Tourism (35 per cent, or $49.7 million), or otherwise had implications for the tourism industry (27 per cent, or $38.3 million). [XV] The largest amounts of funding related to the Growth Strategy for Tourism were within marketing ($22.2 million) and construction ($18.4 million) projects. Most of the funding that supported event projects (67 per cent, or $5.7 million) was also related to the Growth Strategy for Tourism. The extent to which CI programming aims to target the tourism sector and its importance in achieving CI outcomes in relation to other types of projects is not clear in CI documentation.

For outside funding that supported tourism and community infrastructure for recreation and culture, a combined 35 per cent ($64.5 million) supported: infrastructure and equipment purchases for industry-sector related training, research and other activities; infrastructure that enables economic activity such as broadband, sewage/drainage and electrical upgrades; buildings that accommodate commercial space; and, training to enhance business or industry-related skills.

Immediate Outcomes of Investment Projects

Judgment Criteria	Key Findings
There is sufficient evidence to argue the contribution of investment programming to the achievement of immediate outcomes: all projects identify collaborators; the average number of collaborators on a project is 2.7 or greater; 50 per cent of projects have resulted in the increased ability of communities to respond to subsequent opportunities and economic crises; and projects resulted in a response to an economic opportunity or crisis.	According to surveyed clients and case studies: 98 per cent of investment projects indicated collaborators’ involvement. The average number of collaborators is 17, with a median of 5 per project. 90 per cent of investment projects respond to an economic opportunity or crisis; 84 per cent report increased ability to respond to subsequent opportunities or crises; and 79 per cent of projects created economic development infrastructure.

Judgment Criteria

Key Findings

There is sufficient evidence to argue the contribution of investment programming to the achievement of immediate outcomes:

all projects identify collaborators;
the average number of collaborators on a project is 2.7 or greater;
50 per cent of projects have resulted in the increased ability of communities to respond to subsequent opportunities and economic crises; and
projects resulted in a response to an economic opportunity or crisis.

According to surveyed clients and case studies:

98 per cent of investment projects indicated collaborators’ involvement. The average number of collaborators is 17, with a median of 5 per project.
90 per cent of investment projects respond to an economic opportunity or crisis; 84 per cent report increased ability to respond to subsequent opportunities or crises; and
79 per cent of projects created economic development infrastructure.

All lines of evidence show that investment programming leads to immediate outcomes of enhanced collaboration, the creation of infrastructure, and improved capacities. Survey findings below are based on 154 responses (representing 38 per cent of invited survey participants who had obtained funding for investment projects).

Nearly all (98 per cent) of the respondents reported the involvement of collaborators in their initiatives. [XVI] On average, the completed projects reported 17 collaborators per project, with a range of 0 to 1,127. Given the broad range, the median of five collaborators per project is a more meaningful indication of the extent of collaboration.

Respondents reported provincial governments as the most common collaborators (88 per cent), followed by municipal governments (78 per cent), other federal government departments (63 per cent), non-governmental organizations (56 per cent), industry (55 per cent), academic institutions (49 per cent), and Indigenous organizations (37 per cent).

Case studies echo survey findings in that all initiatives included more than five collaborators across a range of organization types, with provincial governments being the most frequent collaborator. [XVII] Furthermore, case studies illustrate the importance of collaborators in achieving results and in the variety of roles they play, whether as funders, as providers of non-financial resources, including expert advice, or as a benefactor.

Survey results suggest that almost all investment projects were developed in response to an economic crisis or opportunity. Ninety per cent of investment clients surveyed reported direct links to an opportunity or crisis. Case studies provide examples of investment projects aimed at developing new industries or mitigating crises related to the economic downturn and the decline of traditional industries.

Case Studies – Investments: Responding to Economic Opportunities and Crises

Several case studies illustrate how, in response to a decline in traditional industries, projects addressed the need to diversify the economy by developing non-traditional sectors in the Atlantic region, such as cranberry production in N.L., e- health in P.E.I., aerospace and defence in N.B. and call-centre support in N.S.

The Celtic Colours festival case study showed how a large festival can address the issue of declining tourism markets, lengthen the tourism season, and build on opportunities related to the unique regional culture and local geography. In 2012, 55 per cent of festival attendees came from outside Cape Breton.

Most investment clients surveyed (84 per cent) reported an increased ability of communities or organizations to respond to subsequent opportunities or crises. Improved capacity to address subsequent opportunities or crises is facilitated by the skills, leadership and experience gained through working with partners to implement initiatives.

A large majority of respondents (79 per cent) reported the creation of physical or non-physical economic development infrastructure. The result was nearly 100 per cent for construction/infrastructure projects. The majority of case studies were linked to the construction or renovation of a building or the creation of non-physical infrastructure such as marketing a festival that brings tourists from outside the region or supporting an industry association to develop a new sector.

Intermediate Outcomes of Investment Projects

Judgment Criteria	Key Findings
There is sufficient evidence to argue the contribution of investment programming to the achievement of intermediate outcomes: jobs created or maintained; reduced out-migration; creation or survival of businesses; increased sales and revenues for businesses in the region; attraction of investment; increased exports; new markets developed; diversified exports; product development; business skills development; and tourism visitation.	The most frequent impacts of investment projects are increased tourism visitation, the development of new products, and increased sales and revenues for businesses. Over half of proponents surveyed said projects resulted in the creation or maintenance of jobs, led to new financial resources for initiatives and new investments, and/or created or maintained businesses. Investment projects did not often lead to impacts related to exports or reduced out- migration.

Judgment Criteria

Key Findings

There is sufficient evidence to argue the contribution of investment programming to the achievement of intermediate outcomes: jobs created or maintained; reduced out-migration; creation or survival of businesses; increased sales and revenues for businesses in the region; attraction of investment; increased exports; new markets developed; diversified exports; product development; business skills development; and tourism visitation.

The most frequent impacts of investment projects are increased tourism visitation, the development of new products, and increased sales and revenues for businesses. Over half of proponents surveyed said projects resulted in the creation or maintenance of jobs, led to new financial resources for initiatives and new investments, and/or created or maintained businesses. Investment projects did not often lead to impacts related to exports or reduced out- migration.

Investment projects often contribute to the achievement of intermediate outcomes, most notably increased tourism, the development of new products, and increased sales and revenues for businesses. According to survey results, tourism impacts were the most common intermediate outcomes, with a significant majority (79 per cent) of respondents reporting increased visitation – a finding that reinforces the extent of support for tourism suggested by the project analysis. Case studies illustrate how investments in community recreational infrastructure can support tourism, and how support for a major tourism festival can increase external visitation.

Case Studies – Investments: Increased Sales and Revenues for Businesses

In 2011, the Celtic Colours International Festival in Cape Breton resulted in $472,837 in direct ticket sales and an estimated $8.1 million in audience expenditures in the region on accommodations, food, art, transportation and retail.

Upgrades to the Yarmouth Mariners Centre contributed to increased facility rental revenues, from $50,000 in 2009 to $125,000 in 2012. The project also contributed to Yarmouth being able to host the World Hockey Junior A Challenge in 2012 and 2013; this event generated $2.6 million of economic activity in the Yarmouth area in 2012 alone.

A large majority (78 per cent) of respondents reported increased sales and revenues for businesses. Again, case studies illustrate how the development of an industry, the creation/expansion of a tourism event, or community infrastructure can increase revenues to businesses.

A significant majority (78 per cent) of respondents also reported the development of a new product. These were often new tourism products such as tours, festivals and other events.

According to survey results, jobs were created or maintained by 70 per cent of investment projects. Of the 78 respondents that reported job figures, 33 per cent created one to two jobs and 30 per cent created three to five jobs. With respect to supporting continued employment, 36 per cent of respondents reported that their projects contributed to maintaining one or two jobs and 26 per cent of projects reported contributing to maintaining three to five jobs.

Case Study – Investment: Creating or Maintaining Jobs and Businesses

The Membertou Economic Benefits Office was supported to help facilitate the participation of Indigenous businesses and individuals in the Sydney tar pond cleanup exercise.

Approximately $49 million in contracts for the clean-up exercise were awarded to six businesses, which created 69 jobs. The tar pond clean-up was successfully completed and the organization has moved on to support Indigenous business and job growth in relation to other projects.

Survey findings also suggest that close to two-thirds (60 per cent) of investment projects resulted in the creation or maintenance of businesses in communities. Of the 49 respondents reporting business impacts, about 50 per cent stated creating one to two businesses and a further 10 per cent reported three to five businesses; 25 per cent indicated that no businesses were created. Almost half of the 55 respondents reporting that their project had contributed to the maintenance of businesses indicated that one to two businesses were maintained; 27 per cent indicated that three to five businesses were maintained.

Over two thirds (67 per cent) of respondents reported obtaining new financial resources for initiatives as a result of their project, which may have been from other levels of government or private sources. Over half (61 per cent) of respondents reported that their project led to obtaining new investments in communities or regions. Case studies demonstrated how multi- purpose community centres can result in the building of new hotels or other infrastructure, adding to the economy of the communities. In the case of a multi-purpose centre in Grand Falls, a new hotel and gas station were developed as well as a new residential sub-division, expanding the tax bases and amenities for the region.

Case Study – Investment: Developing Business Skills

In response to industry research suggesting a need to further develop tourism products/experiences in Atlantic Canada to meet market demand, ACOA supported a project by the Gros Morne Institute for Sustainable Tourism to deliver workshops on experiential tourism, best practices in the use of online tools, and sustainable tourism practices to 1,249 tourism operators across Atlantic Canada.

Facilitators with tourism expertise in each region were trained to deliver the workshops and to tailor them using local content, which built capacity for future support to the industry.

In a follow-up survey, 67 per cent of participants reported using the knowledge to make changes in marketing, use of technology or product development.

Almost half of the investment clients surveyed (48 per cent) reported the development of new markets. Case studies of the cranberry industry in Newfoundland and Labrador and the aerospace and defence industry in New Brunswick provide examples of how projects can impact the development of new markets. The case study of the Celtic Colours International Festival also demonstrated how it developed new markets for local Celtic musicians through links to other international festivals.

According to a detailed project analysis, 49 of the 749 projects were dedicated to building business skills in the areas of management, productivity, market readiness, innovation and commercialization. The case study of the Gros Morne Institute for Sustainable Tourism provides an example of how ACOA contributes to building business skills within a given sector.

Outcomes related to exports and to reducing out-migration were not as commonly achieved as others. Only 28 per cent of investment respondents indicated success in diversifying exports (55 per cent stated it was not applicable), and only 23 per cent indicated success in increasing exports (51 per cent stated it was not applicable). Only 27 per cent of respondents reported reducing out- migration (40 per cent stated it was not applicable). These findings suggest that outcomes relating to exports and out-migration might not be directly relevant for investment projects.

4.3 Unintended Outcomes of Planning and Investment Programming

Judgment Criteria	Key Findings
Not Applicable	There were three main unintended outcomes of programming that should be considered in program descriptions as well as project viability assessment processes: enhanced community capacity and quality of life; spinoffs; and negative consequences for organizations and communities.

Judgment Criteria

Key Findings

Not Applicable

There were three main unintended outcomes of programming that should be considered in program descriptions as well as project viability assessment processes:

enhanced community capacity and quality of life;
spinoffs; and
negative consequences for organizations and communities.

The evaluation identified three categories of unintended outcomes of CM and CI programming: increased community capacity and quality of life; economic spinoffs; and, negative consequences. These outcomes are not included in the existing program logic models but are identified in the CED Conceptual/Analytical Model. They are important outcomes to consider in clarifying the types of projects funded and the expected results of programming as well as in mitigating risks.

Case Studies – Investments: Quality of Life and Spin-offs

Projects contributing to multi-purpose centres in Yarmouth, N.S., Grand Falls, N.B., and Shediac, N.B., showed that infrastructure supports local businesses as well as cultural and sporting activities in the community and is seen to contribute to a greater standard of living.

These investments can be strategic in retaining labour force provided that there is sufficient employment in the area to meet income needs and allow for resident use. They also result in spinoff revenues generated for restaurants, catering companies, accommodations, etc.

Informants emphasized that the improved amenities and services increased pride among residents and hope for the future of their communities and their ability to overcome economic challenges.

Community Capacity and Quality of Life

Case studies and key informant interviews frequently highlighted outcomes related to improved community capacity and quality of life from having better amenities. Improved community resources, including new multi-use facilities or recreational centres, can be used to retain and attract residents as well as contribute to feelings of community connectivity, pride and hope.

Spinoffs

According to case studies and key informant interviews, there were several examples of economic spinoffs as a result of planning and investment programming:

As previously mentioned, important community infrastructures can support existing and new businesses such as gas stations, restaurants, hotels and other services. New businesses also contribute to the municipal tax-base, which can be used to increase the quality of services or decrease the cost to residents.
Two case studies in particular faced challenges due to the lack of skilled labour. The proponents turned the challenge into an unintended positive outcome by working with partners to develop new or local training programs to meet industry demand.
Several successful projects are being replicated as best practices. The Membertou Economic Development Office model is being considered for more Indigenous communities in Nova Scotia and in British Columbia. The GMIST tourism operator training program is being replicated in Manitoba. According to key informants from a leading national tourism organization, GMIST training is a best practice.

Negative Outcomes

Case studies and interviews suggested that weak project planning or organizational capacity can result in the inability to meet expected outcomes, and in financial stress for the proponent and partners. For example, case studies indicated that events that did not meet expected ticket sales or community buildings that did not meet rental targets caused organizations to take on additional debt or to close entirely.

There was also evidence of stress on volunteers with greater than manageable expectations in terms of time and duties. Availability of volunteers and volunteer burnout is becoming a greater issue with an aging and declining rural population.

Case Study – Investments: Weak Planning and Management Capacity Affecting Project Outcomes

A case study that examined the construction of a multi-purpose centre shows that outcomes were negatively affected by weak board capacity, poor planning and volunteer burnout. The structure did not meet the needs of the target market. Since it did not generate the revenues expected, its continued maintenance resulted in further stress to the non-profit organization and its volunteers.

4.4 Roadmap for Linguistic Duality Initiatives: Achievement of Outcomes

As previously mentioned, between 2008-2009 and 2011-2012, ACOA CM and CI staff approved 34 projects for $5.19 million in approved funding in support of the EDI. A 2012 Summative Evaluation of projects delivered as part of the EDI reported the achievement of the following immediate outcomes: development of new expertise through innovation; diversification of activities; partnerships; and increased support for small businesses. The evaluation also concluded that the achievement of intermediate results was strong and demonstrated significant capacities built, and enterprises and communities developed. Unintended outcomes included the development of closer relationships with OLMCs and higher than expected leveraging from other organizations. [42]

The 2012 evaluation of the $10 million New Brunswick francophone immigration initiative funded under the Roadmap reported that the initiative conducted promotional and awareness activities with employers, communities and the general public. The report stated the initiative resulted in greater than expected immigration numbers to date, and is well-positioned to achieve the expected outcome of increasing immigration though the time frame did not allow for a full assessment of outcome achievement. [43]

4.5 Overall Achievement of CED impacts

Overall, the evaluation provides evidence that the capacity and asset base of communities is strengthened as a result of planning and investment funding offered by ACOA and its partners. Planning and investment programming provides access to capital that contributes to planning for economic development and creating important economic infrastructure and other supports. However, the evaluation cannot speak to the overall impact of this strengthened asset base on the economic viability and sustainability of communities.

The evaluation lends support to the activities, impacts and contextual factors outlined in the CED Conceptual/Analytical Framework (Appendix C). The framework illustrates the importance of the capacity and asset base from which communities draw in order to: respond to economic opportunities and threats; increase economic stability and sustainability; and, improve the quality of life of residents. Furthermore, the evaluation confirms many of the contextual challenges or barriers to the achievement of outcomes, detailed in the next section as well as in the section on program relevance.

4.6 Barriers to the Achievement of Planning and Investment Outcomes

Judgment Criteria	Key Findings
Evidence of factors that are impeding the success of the CM and CI sub-programs are known, and mitigation strategies are implemented where applicable.	Three key factors impeded the achievement of planning and investment outcomes: capacity and business skills: project proponent: governance, planning and risk management, leadership, business skills, partnerships, project management, volunteers ACOA: insufficient proposal assessment, gaps in knowledge sharing across teams contextual issues, most notably the economic downturn, community and industry assets and distance from markets for many regions funding issues, including the availability of funding, project costs and approval processes. While some mitigation strategies for management capacity and funding barriers were identified, they were not consistent across the Agency.

Judgment Criteria

Key Findings

Evidence of factors that are impeding the success of the CM and CI sub-programs are known, and mitigation strategies are implemented where applicable.

Three key factors impeded the achievement of planning and investment outcomes:

capacity and business skills:
- project proponent: governance, planning and risk management, leadership, business skills, partnerships, project management, volunteers
- ACOA: insufficient proposal assessment, gaps in knowledge sharing across teams
contextual issues, most notably the economic downturn, community and industry assets and distance from markets for many regions
funding issues, including the availability of funding, project costs and approval processes.

While some mitigation strategies for management capacity and funding barriers were identified, they were not consistent across the Agency.

Case studies and client survey data were the primary source of information relating to barriers and facilitators of project success. Key informant interviews, particularly with ACOA staff and management, also identified internal challenges. Both barriers and facilitators fell into three categories: capacity and business skills, contextual factors, and funding. Mitigation strategies were not consistently implemented across the Agency.

Capacity and Business Skills

Case studies and key informants identified issues related to capacity and business skills on the part of the project proponent or the Agency. For the project proponents, weak governance structures, planning and risk management practices, leadership, business knowledge and skills, including project management, lack of strategic partnerships and limited volunteer capacity led to a greater risk of not achieving project outcomes. ACOA interviewees indicated that governance training was provided as needed to organizations experiencing difficulties with their board or overall management. However, training was generally on a case-by-case basis and was not carried out by all regions.

ACOA’s capacity to assess projects also affected project outcomes. Several case studies provided evidence that inadequate project viability assessment and premature project approval can contribute to less successful projects and weaker outcomes. Key informants highlighted the importance of sharing information and looking for synergies with the other ACOA units, PAC and ED . Early concept vetting committees, which include representatives from other units, and related processes have been implemented in some regions and show promise in mitigating some internal management challenges.

Contextual Factors

The evaluation identified numerous contextual factors through survey data, key informant interviews and case studies that can impact the achievement of project outcomes:

The economy, particularly the global economic downturn, was a frequently cited barrier to achieving project outcomes. The economy affected the amount of sales and business growth achieved.
Distance from markets is another important contextual issue for Atlantic Canada. Case studies related to tourism or the development of specific industries noted distance from markets as a challenge for achieving outcomes.
Communities that lack important assets such as transportation infrastructure (e.g. airports, airline services or ferries) face particular issues in developing or maintaining industries and attracting tourists.
As previously stated, out-migration and an aging population impact the availability of volunteers for non-profit organizations; this also can lead to volunteer burnout as some feel obligated to dedicate more effort or continue in volunteer roles for longer periods than desired.
Case studies showed that a lack of industry assets can also lead to weaker achievement of project outcomes. For new, specialized industries such as aerospace and defence, the smaller number of companies in Atlantic Canada leads to fewer members for associations and weakened ability to effectively develop relationships with large multinationals. The tourism industry is seasonal and has a lower profit margin, affecting pay scales and the availability of resources for staff or management training.

Case Study – Investments: Increasing Competitive Advantage

In the case of the renovation of a large historic building to develop the e-Health industry in Summerside, P.E.I., the lack of local amenities, including a local airport with regular flights, and the absence of training in the community were reported barriers to project success.

ACOA worked with the province and other partners to attract new companies to the building and provide refit costs. The Agency also collaborated to bring a specialized training program through Holland College to Summerside, mitigating issues related to lack of skilled workers.

The Agency can play a role in levelling the playing field for business attraction and industry sector growth. Some case studies show that ACOA has considered the lack of particular community and industry assets in its project development and review processes to a certain extent (see example in above box).

Funding

As stated in section 3.0, funding was the most important barrier to project success according to surveyed clients. The availability of funding is often tied to the economy and was frequently linked to the capacity of provincial governments and other funders to contribute to projects.

Delays in funding approval processes can also influence project success. Key informants spoke of cost overruns that can occur, for example, if construction is delayed to winter months or if construction costs increase over the delay period.

4.7 Lessons Learned and Best Practices in Delivery

Judgment Criteria	Key Findings
Not Applicable	Key lessons learned and best practices: The strong assessment of proponent organizational capacity, funding and business plans by ACOA decreases risks and improves the achievement of outcomes. Short-term programming such as the EAP and CIIF can cause confusion among clients and other stakeholders about regular programming parameters. Short-term funding can also deplete the funds available from partners and potential project proponents, affecting the number and quality of project proposals through the regular planning and investment programming. Approval delays are detrimental to projects in terms of missing opportunities and facing cost overruns. Self-sustainability may not be a reasonable expectation for all projects, particularly those that support groups or communities that experience greater financial or other challenges and are unable to cover costs. Regional approaches to planning and investment projects align with the notion of functional communities and offer opportunities to develop community partnerships and make the best use of limited funds.

Judgment Criteria

Key Findings

Not Applicable

Key lessons learned and best practices:

The strong assessment of proponent organizational capacity, funding and business plans by ACOA decreases risks and improves the achievement of outcomes.
Short-term programming such as the EAP and CIIF can cause confusion among clients and other stakeholders about regular programming parameters. Short-term funding can also deplete the funds available from partners and potential project proponents, affecting the number and quality of project proposals through the regular planning and investment programming.
Approval delays are detrimental to projects in terms of missing opportunities and facing cost overruns.
Self-sustainability may not be a reasonable expectation for all projects, particularly those that support groups or communities that experience greater financial or other challenges and are unable to cover costs.
Regional approaches to planning and investment projects align with the notion of functional communities and offer opportunities to develop community partnerships and make the best use of limited funds.

Lessons Learned

Key informant interviews and case studies reveal important lessons learned and best practices for planning and investment programming that can be considered for program improvement. Perhaps the most important lesson learned is the need for a strong, consistent assessment of project viability and risk from the outset in order to reduce risks and improve outcomes. There are promising regional tools and processes for project viability assessment, such as the use of project concept vetting committees.

Initiatives aimed at building economic development capacity for minority groups such as Indigenous and less lucrative or emerging sectors may have fewer assets to contribute and require more support to implement economic development projects, particularly if they cannot obtain much support from other funding organizations. If these sectors or groups are determined to be Agency or regional priorities for development, their existing assets and the availability of other funding or lack thereof should be considered in determining the extent of support that ACOA will provide, and whether operational/administrative activities should be supported.

With respect to EAP programming, key informants identified two lessons learned. First, while it offered many opportunities to communities, EAP projects also often depleted potential project proponent and partner funds, affecting their ability to engage in other economic development projects. Second, because EAP programming was managed by the Agency and had different funding requirements and parameters from regular programming, it caused some confusion among clients and partners in terms of what could be funded through core programs.

Best Practices

According to program key informants, collaboration of RDAs in the development of new programs was a best practice that resulted in efficiencies through sharing of responsibilities attached to the development of performance measurement strategies, TB submissions and sharing of best practices. Another best practice was the use of ACOA’s existing terms and conditions for administering CAF. According to key informants and a literature review, an important best practice is striving for regional approaches to CED in order to maximize the impacts and reach of investments, especially in the context of demographic changes and population movements. Several case studies demonstrated the benefits of communities collaborating on infrastructure such as tourism events and multi-purpose centres. These initiatives benefited from shared resources, including funding, infrastructure and effort, while serving a greater number of people in many cases. Supporting Atlantic-wide delivery of a high-quality business training course can also be seen as a best practice when it allows for cost savings in course development and it meets a common need.

4.8 Adequacy of Performance Measurement

Judgment Criteria	Key Findings
Evidence that CM and CI PSA performance measurement (PM) is adequate and effective in reporting on the achievement of outcomes. PM information is used by ACOA and ACOA-funded organizations to support decision making. Performance information is available for use in the evaluation.	PM was not adequate or effective in reporting on the achievement of outcomes. PM information was used for DPR reporting, including the: number of projects, amount of investments, number of plans completed and number of partners involved in planning projects. The Results Management Tracking System (RMTS) is not used consistently, though it is promising as a tool to track outcomes. PM is used to some extent by ACOA management to support decision making, but aggregate data was limited. The evaluation unit used DPR data and financial information as well as project final reports when available. Project coding through QAccess was limited in providing information on the types and scope of projects funded.

Judgment Criteria

Key Findings

Evidence that CM and CI PSA performance measurement (PM) is adequate and effective in reporting on the achievement of outcomes. PM information is used by ACOA and ACOA-funded organizations to support decision making. Performance information is available for use in the evaluation.

PM was not adequate or effective in reporting on the achievement of outcomes. PM information was used for DPR reporting, including the: number of projects, amount of investments, number of plans completed and number of partners involved in planning projects. The Results Management Tracking System (RMTS) is not used consistently, though it is promising as a tool to track outcomes. PM is used to some extent by ACOA management to support decision making, but aggregate data was limited. The evaluation unit used DPR data and financial information as well as project final reports when available. Project coding through QAccess was limited in providing information on the types and scope of projects funded.

PM data is used to some extent by ACOA to aid program management. ACOA uses high-level dashboard data to monitor program activity such as the number of projects and funding amounts. Management uses QAccess queries and project reports (when available) to establish project precedent in terms of activities and/or outcomes and to inform decision making. However, key informants suggested this does not adequately support strategic program management. PM data collected for DPRs is limited to the number of plans and number of partners for CM programming, and to the number of projects and amounts funded for CI programming. There is limited reliable, aggregate-level data on the nature of approved projects and their outputs, and the achievement of project outcomes.

More consistent use of available QAccess fields would enable the capturing of useful information for strategic program management, such as the number of strategic/proactive investments and projects that support cross-Agency priorities such as tourism, marketing and skills development. However, existing QAccess fields are limited in capturing some important project types such as infrastructure aimed at community amenities with recreational or cultural ties; infrastructure aimed at supporting certain industry sectors; and investments in events that support economic development.

Though some financial data were available and used in the evaluation, particularly from ACOA’s GX and QAccess systems, substantial resources had to be invested in the production of reliable project information due to issues with project coding, inconsistent project summary form content, the inconsistent collection and uploading of final reports and other limitations. ACOA’s RMTS, which will allow account managers to track outcomes, holds promise for the availability of outcome data but is not yet fully implemented.

The majority of clients surveyed reported that they were collecting and monitoring performance data (94 per cent) and using the information for decision making (87 per cent). Several case studies demonstrated PM data was being used for reporting economic impacts and other outputs and outcomes to funders.

5. Findings: Performance – Efficiency and Economy

The evaluation assessed efficiency and economy by examining (1) the utilisation of resources, (2) evidence of mechanisms that support efficiency and economy, and (3) consideration of alternative delivery mechanisms and best practices that resulted in efficiency and economy gains for ACOA.

5.1 Efficient Utilisation of Resources

Judgment Criteria	Key Findings
Delivery costs across regions reasonably compare to one another within the context of factors affecting regional delivery.	Delivery costs are proportionate to the amount of funding and number of projects delivered. ACOA was efficient in delivering EAP programming. Amount of G&C delivered by salary dollar and by operational dollar compared reasonably between regions.
ACOA has in place mechanisms to ensure that the most efficient and economical means are being used to administer the programming. Internal tools and mechanisms allow for strategic decision-making in the selection of projects that result in critical economic infrastructure (or non-physical infrastructure) being developed.	Mechanisms that support efficient and economical delivery include regional strategic planning exercises to help focus investments, completion of feasibility studies prior to investments and leveraging of external funds, which has increased since the previous evaluation. ACOA does not have a reliable process in place to regularly track types of investments funded, which challenges its ability to manage the program strategically and efficiently. The extent of coordination between units or programming areas that supported similar projects was unclear. Most regions have increasingly focused on developing and approving projects with the greatest potential for achieving economic impacts. Tools and mechanism that support these efforts include the development of regional community investment plans and concept vetting committees. The use of such tools and mechanisms is not consistent across regions.

Judgment Criteria

Key Findings

Delivery costs across regions reasonably compare to one another within the context of factors affecting regional delivery.

Delivery costs are proportionate to the amount of funding and number of projects delivered. ACOA was efficient in delivering EAP programming. Amount of G&C delivered by salary dollar and by operational dollar compared reasonably between regions.

ACOA has in place mechanisms to ensure that the most efficient and economical means are being used to administer the programming.

Internal tools and mechanisms allow for strategic decision-making in the selection of projects that result in critical economic infrastructure (or non-physical infrastructure) being developed.

Mechanisms that support efficient and economical delivery include regional strategic planning exercises to help focus investments, completion of feasibility studies prior to investments and leveraging of external funds, which has increased since the previous evaluation. ACOA does not have a reliable process in place to regularly track types of investments funded, which challenges its ability to manage the program strategically and efficiently. The extent of coordination between units or programming areas that supported similar projects was unclear. Most regions have increasingly focused on developing and approving projects with the greatest potential for achieving economic impacts. Tools and mechanism that support these efforts include the development of regional community investment plans and concept vetting committees. The use of such tools and mechanisms is not consistent across regions.

5.1.1. Delivery Costs

Delivery costs were generally proportional to the amount of programming delivered and compared reasonably across regions over the evaluation period.

In order to provide the most accurate picture of expenditures specific to CM and CI sub-programs, the cost of internal services, Treasury Board-funded costs and other statutory and miscellaneous expenditures have been excluded from subsequent analyses. These costs amount to $39.5 million over the four-year period from 2008-2009 to 2011-2012, or an average of $18 million a year.

Expenditures for CM and CI programming over the evaluation period amounted to $411.5, with $380.3M (92 per cent) in G&Cs, $6.5M (2 per cent) in O&M costs and $24.7M (6 per cent) in salaries. Figure 2 represents program expenditures by region, with ACOA NB showing the largest amount of total expenditures, followed by NL, NS, PEI and CB.

Figure 2: Community Mobilization and Community Investment Expenditures by Region (2008-2009 to 2011-2012)

(D)

Source: GX Financial Data, August 2012; ECBC financial records, December 2013.

The amount of G&Cs delivered varied by region due to factors such as regional demand, geographic coverage, population served, and regional priorities. When considering all G&Cs expenditures including the EAP , ACOA NB delivered the most programming ($108.8M), followed by NL ($99.2M), then NS ($76.6M), PEI ($53.3M), CB ($39.5M) and ACOA Head Office ($2.8M).

Salary expenditures associated with delivering the programming compared favourably with the previous evaluation, representing a slight decrease from the previous 6 year period (5.8 per cent vs 6.2 per cent) due in part to the additional delivery of EAP initiatives, the Recreation Infrastructure Canada (RInC) and CAF. These costs supported activities such as working with clients on proposal development, proposal assessment and approval processes, processing financial claims, monitoring, as well as other activities.

Annual salary costs per region were reviewed (see appendix H for more detail). A general increase in salary expenditures in 2009-2010 is attributed to account manager and support staff time dedicated to delivering EAP initiatives. The ratio of G&C funding delivered per salary dollar helps interpret regional differences. These ratios compared reasonably among ACOA regions with CB delivering $18.29 in G&C per salary dollar, followed by $17.90 for NB, $15.70 for NS, and $14.10 for NL. PEI delivered $26.60 ; this region covers less geography and operates with a smaller staff contingent, making for a reasonable comparison. The calculation of G&C per salary dollar was not considered for HO recognizing that the majority of staff activities do not focus on the direct delivery of G&Cs but are dedicated to program governance, coordination and reporting along with other corporate functions that support the programming such as internal human resource and finance services.

Beyond salary expenditures, other operational costs were reasonably consistent across regions. Figure 3 presents operational spending by ACOA region, including costs associated with communications, travel, and professional services such as conducting environmental assessments required for infrastructure projects. Ratios of G&C dollars delivered per operational dollar help to contextualize the regional differences. For the most part, ACOA regions compare reasonably well with CB delivering $108.36 in G&Cs per operational dollar spent, followed by NB ($85.10), PEI ($69.55), and NS ($54.40). NL is lowest ($42.64), in part due to the increased cost of travel and professional services associated with its greater geographic coverage. The specific expenditures by type of operational activity for CB region were not available but these included activities such as travel for client monitoring and training, rentals, and other activities that supported the delivery of the programming.

Figure 3: Operational Spending per Region

(D)

Source: GX Financial Data, August 2012;ECBC financial records, December 2013.

5.1.2 Mechanisms that Support Efficient and Economical Delivery

Multiple mechanisms currently exist to support efficient and economical program delivery. These planning, governance and delivery mechanisms were often aimed at selecting and supporting projects with greater potential of achieving economic impacts. The use of these tools and mechanisms varied between regions.

Planning mechanisms:

Most ACOA regional offices developed investment strategies that described contextual issues and prioritized activities and areas for investment, in support of strategic decision making. ACOA NB region developed northern and southern NB plans. NL and NS regions developed CI investment plans and PEI identified priorities through its annual planning processes. A few regions coordinated investment planning with provincial governments - ACOA PEI region coordinated with the Province on its rural diversification program and ACOA NL held meetings with the Province to discuss shared priorities.
Case studies provided evidence that project-level planning sometimes incorporates needs assessments or asset mapping to maximize the use of pre-existing resources and focus funding resources where most needed. Projects also carried out feasibility studies as a means to ensuring strong planning prior to making larger scale investments.

Governance mechanisms:

Annual Community Development meetings with all levels of management played an important role in coordinating delivery.
Pan-regional management committees coordinate delivery of ICF funding.
NB and NS used committees to engage in early consideration and vetting of all projects to identify those with the strongest potential for sustainable economic impacts, increase awareness of projects and key issues between managers and support consistency in decision-making.
Regular budget review processes also promoted efficient program governance by allowing funding to be redistributed based on assessment of need.

Delivery mechanisms:

There was increased focus on projects described as transformative and having more sustainable economic impacts. Data is not yet available to support actual gains in efficiency.
The collaboration between ACOA and other RDAs in the delivery of the EAP created efficiencies by dividing the responsibilities attached to the development of performance measurement strategies, TB submissions and sharing of best practices.
ACOA delivered the EAP programming using the terms and conditions of existing G&Cs and without additional operational resources, demonstrating efficiency and economy. However, as previously discussed, as resources shifted to the delivery of the EAP , there was a decline in the number of core CI projects.
Programming was delivered through geographically-based teams in NS, NL and NB, leading to improved coordination and reduced travel costs. By assigning ACOA CD staff to specific geographic areas, efficiencies can be gained from their specialized knowledge of a region and its key stakeholders for economic development. In terms of reducing travel costs, staff can capitalize on the opportunity to liaise with clients from all aspects of CD programming from that area in fewer visits, rather than having staff associated with different aspects of CD programming travel separately and at different times.
The Kaizen continuous improvement process in NB for the delivery of ICF resulted in the establishment of an early project review committee, development of program proposal guides for clients to increase the quality of applications and the standardization of project summary forms to improve the quality and consistency of information presented in the document. According to an internal assessment, project evaluation time was reduced by 50 per cent (8 days) as a result of these changes.
The program levered substantial funding from other organizations, which contributed to its efficiency. For ACOA’s core CM and CI programming, ACOA contributed $226.9M of a total $713.2M in project cost, leveraging $486.3M from other organizations. On average, every ACOA dollar invested levered an additional $2.14 from other organizations, representing a substantial increase from the previous evaluation, which reported leveraging of 0.68$ for every ACOA dollar invested for planning and mobilization activities, and 0.95$ for every ACOA dollar of investment activities.

5.2 Alternative Modes of Delivery for Increasing Efficiency and Economy

Judgment Criteria	Key Findings
Program management considers alternative modes of delivery outside the organization	Alternative ways of increasing efficiency and economy were considered and some were implemented, including the use of pan-regional management committees and efforts to fund projects with greater economic impacts. However, key informants caution that proactive development requires more resources and can carry greater risks related to ACOA’s increased influence on projects, and increased perceived accountability for project results The decision to discontinue operational funding of REDOs reduces overlap in services with other organizations and is expected to allow ACOA to provide more direct client services. A number of suggestions were made to improve efficiency including increased collaboration between ED , CD and PAC, and increasing client awareness of programming goals to improve proposal quality.
The organization considers its own best practices in implementing alternative approaches for increased efficiency. Best practices were identified and implemented.

Judgment Criteria

Key Findings

Program management considers alternative modes of delivery outside the organization

Alternative ways of increasing efficiency and economy were considered and some were implemented, including the use of pan-regional management committees and efforts to fund projects with greater economic impacts. However, key informants caution that proactive development requires more resources and can carry greater risks related to ACOA’s increased influence on projects, and increased perceived accountability for project results

The decision to discontinue operational funding of REDOs reduces overlap in services with other organizations and is expected to allow ACOA to provide more direct client services.

A number of suggestions were made to improve efficiency including increased collaboration between ED , CD and PAC, and increasing client awareness of programming goals to improve proposal quality.

The organization considers its own best practices in implementing alternative approaches for increased efficiency. Best practices were identified and implemented.

Alternative ways of achieving outcomes were considered and some actions were taken to increase efficiency and economy. The most significant change in delivery is ACOA’s decision to discontinue operational funding to REDOs for regional planning and mobilization is expected to result in further efficiencies but again, the shifting landscape in key players carries a risk of reduced capacity to coordinate and implement initiatives. Also as stated in the previous section, pan- regional management committees allowed for exchanging best practices and coordinating delivery of CI projects. Several other practices were considered and to some extent implemented to help maximize economic impacts of projects such as harnessing resources of ED programming and PAC units, strategic planning efforts and concept vetting processes. Key informants cautioned that efforts to develop projects with greater potential for economic impacts require more internal resources and time during project development stages. In particular, they spoke of the time needed to for both internal and external coordination with key players and capacity building with clients. Also, proactive project development may hold greater risks in terms of ACOA’s perceived accountability for increased influence over the nature of projects, particularly considering ACOA’s incremental role in obtaining funds from other organizations.

Key informants suggested possible improvements to increase efficiency, including: promoting program goals to potential client groups to maximize proposal quality; further collaboration between units for project development; and better assessment of proposal/concept viability and client capacity to implement and sustain initiatives where applicable. Instances were cited where lack of such scrutiny allowed for investments in some weaker projects, negatively affecting efficiency and economy. It was noted guidelines were being developed to improve project assessment, but were not fully implemented. Other suggestions included the need to streamline ICF approval processes to the extent possible, to reduce approval delays. As previously mentioned, delays in project assessment and approval have resulted in missed opportunities for promising initiatives and/or increased cost (e.g. winter construction). Also, the availability and use of better information on program activities and outputs could enhance strategic decision making, as stated in section 3.

6.0 Conclusions and Recommendations

Based on the findings presented, the evaluation identified 7 conclusions:

ACOA’s CM and CI programming responds to a continued need and is aligned with GoC priorities (i.e. EAP , Roadmap for Linguistic Duality), and the Agency’s priorities and mandate (i.e. economic competitiveness and growth; ACOA’s Growth Strategy for Tourism). ACOA is aware of changing CED needs; some adjustments to CM and CI programming were made to address changing needs including the cessation of the REDO model and efforts to support projects with greater economic impact potential.
ACOA CM and CI programming complements rather than duplicates that of other partners. However, a collaborative, coordinated and strategic approach among partners is critical to the achievement of CED outcomes. ACOA works alongside a variety of stakeholders (who often co-fund projects) towards these outcomes. The dissolution of the REDO model brings about a necessary period of transition as ACOA and CED partners realign resources and clarify roles and responsibilities. During the period of transition, there is a risk of capacity gaps for regional-level planning or other economic development activities.
ACOA planning and investment program funding was incremental to the implementation of projects, to obtaining investments from other funding partners, and to achieving outcomes. Planning projects generally achieved immediate outcomes by increasing CED skills and leadership, influencing decision-making and leading to concrete actions for economic development. It was not possible to conclude on the overall achievement of intermediate outcomes of planning due to low survey representation of projects with completed plans and strategies. However, case studies supported the importance of thorough project planning for greater economic development impacts. Investment projects generally achieved immediate outcomes such as creation of infrastructure for economic development and increased capacity for responding to future opportunities. Intermediate outcomes included increased tourism visitation and sales and revenues for businesses, as well as the development of new products. There was evidence of modest job creation and maintenance as well as modest business creation and survival.
A third of investment funding supported the development of infrastructure for community, recreational and cultural activities. These investments can be useful in providing amenities that attract and retain residents/labour force, where sufficient employment opportunities exist. The role and relative importance of these activities for achieving CI outcomes is currently not well articulated in program documentation. Similarly, 35 per cent of funding directly supported tourism, but the extent to which CI programming aims to support tourism relative to other sectors or other types of activities is not clear. Outside community/cultural/recreational and tourism-related projects, a combined 35 per cent of investment funding supported: infrastructure and equipment purchases for industry-sector related training, research and other activities; infrastructure that enables economic activity such as broadband, sewage/drainage and electrical upgrades; buildings that accommodate commercial space; and, training to enhance business or industry-related skills.
An analysis of facilitators and barriers to the achievement of outcomes as well as lessons learned demonstrates the importance of considering management/delivery factors (i.e. internal – project viability assessment; incorporation of knowledge from ACOA PAC function; external – client skills and capacities), contextual factors (i.e. global economic decline, deteriorating infrastructures), and funding (i.e. availability and timely approval processes). While there was some evidence of the use of strategies to mitigate barriers, it was not clear that knowledge related to common barriers to achieving outcomes is used in the selection or monitoring of projects.
ACOA management uses some performance measurement data pertaining to expenditures and number of projects by transfer payment program to support decision-making. Some PM data was used in the evaluation (e.g. DPR , Q Access, GX), however a detailed analysis of all projects was required to produce information on activities funded. There continues to be challenges with the adequacy of information available on activities, outputs and outcomes for individual projects and at the aggregate level; these challenges present barriers to strategic and efficient program management.
ACOA’s CM and CI programming was delivered in a cost-effective manner. Internal costs are proportionate to the corresponding level of funding involved and number of projects delivered. Leverage of other funding has increased since the previous evaluation. ACOA was efficient in delivering substantial EAP programming without additional dedicated resources; however, this resulted in fewer projects related to ACOA’s core CM and CI programming being delivered.
Several regions have some promising practices that aid in the selection of investment projects that achieve greater economic impacts such as well-informed investment strategies and concept vetting committees. Proactively developing projects with greater impact potential holds promise in increasing selection and funding of projects that maximize economic development impacts but may require more internal resources. Risks associated with increased influence over funded initiatives could emerge.
Some gaps exist in the consistent application of strong proposal viability/risk assessment processes to reduce risks. The extent of coordination between tourism, business skills development, trade and innovation projects with other similar ACOA programs was unclear, which may create inefficiencies in achieving Agency objectives relating to those areas. Currently, there is a lack of consistency in tracking performance reporting and evaluation of these projects across the Agency.

Overall, the evaluation’s recommendations are aimed at improving programming by balancing a consistent, targeted approach with required flexibility; improving the availability and access of information and knowledge to manage for results; and building on existing and new relationships, expertise and intelligence to narrow the gap between ACOA’s community economic theory and practice, as implemented through the CM and CI sub-programs.

Recommendation 1:

The framework should outline the Agency’s approach to planning and investment programming (including main project types and expected outcomes), while acknowledging regional context, plans, priorities and best practices. It should also describe the Agency’s approach to engage, coordinate and collaborate both internally and externally (partners, subject matter experts) in the achievement of programming outcomes.

The following is a list of considerations in developing the framework that build on evaluation findings:

Particular project types to consider include projects aimed at facilitating transformational or longer term, economic impacts; projects in support of ACOA’s Growth Strategy for Tourism, planning, and community/recreational/cultural infrastructure.
Important partners include provincial governments, other federal government departments, municipalities, potential project proponents, community organizations and knowledge/content experts within or beyond Atlantic Canada.
Consider regional contexts, assets, needs and existing investment plans. Integrate best practices such as strong project planning and regional approaches to economic development in the context of declining rural populations and capacity (e.g. consideration of functional economic regions and collaboration between communities).
Given the programming’s decentralized delivery and regional differences in context, assets and needs, the framework should be developed collaboratively, and updated on a regular basis. The framework should be communicated and its use as a communication tool should be encouraged.

Recommendation 2:

In support of results-based management, decision-making, and the strategic coordination of initiatives that cut across programming areas (e.g., business skills development, tourism, trade, innovation), ACOA should ensure that quality and timely information on activities (i.e. main project types), and outcomes are systematically collected on file and in electronic systems.

While new tools may be needed, consideration should be given to consistent implementation and use of existing processes, tools and systems (i.e. PSFs, Q Access coding, results tracking, collection of final reports, 2-year follow-up).

It is expected that the systematic collection and reporting on activities and outcomes of CI initiatives that cut across programming areas will help maximize synergies and minimize the risk of under-representing Agency efforts during evaluations and other performance reporting exercises.

Recommendation 3:

The process/tool should assess the likelihood of achieving outcomes (e.g. market response, use of service or infrastructure), as well as the human (project management/governance, involvement of the right partners), financial, and infrastructure capacity of the proponent organization or community to conduct the work outlined in the proposal and to sustain the achievement of outcomes and impacts following project completion.

Figure 4: Alignment between CM and CI Evaluation Conclusions and Recommendations

(D)

Appendix A: Program Alignment Architecture Chart

Strategic Outcome	Program	Sub-Program
A competitive Atlantic Canadian economy	Enterprise Development	Innovation
		Entrepreneurship and Business Skills Development
		International Business Development
		Financing Continuum
	Community Development	Community Mobilization
		Community-based Business Development
		Community Investment
		Infrastructure Programming
	Policy, Advocacy and Coordination	Policy
		Advocacy
		Coordination
	Internal Services	Governance and Management Support
		Resource Management Services
		Asset Management Services

Source: 2011-2012 Report on Plans and Priorities.

Appendix B: Evaluation Design and Methods

The mixed method evaluation design was selected in consideration of:

the program theory (outcomes and impacts sought, timeframes for capturing information on outcomes);
risks associated with the programming and with the evaluation of key issues relevance, effectiveness and efficiency/economy; and,
the judgment criteria developed with key program stakeholders.

The CD Impact Framework suggested that case studies could be an ideal tool for considering program complexity and timeframes for achieving outcomes. Many external factors can impact the achievement of outcomes. Case studies provided an opportunity to study how impacts are achieved with this programming, while considering important contextual factors. Program theory also suggested that the achievement of intermediate outcomes could take longer than four years. Therefore, the evaluation included a follow-up, or update of results on the case studies from the previous evaluation.

The Treasury Board Policy on Evaluation (2009) requires a risk-based approach to evaluation design. That is, the evaluation approach and level of effort should be calibrated to reflect the risks associated with the programming, while meeting minimum standards of rigour related to research design and lines of evidence. The level or risk associated with CM and CI programming was considered through consultations, preliminary document review, and consideration of previous evaluation findings.

Risks associated with program relevance were considered low to medium, given materiality of the programming and changes that have occurred in the programming environment such as the continued decline of rural areas. Relevance was evaluated through document review and interviews with stakeholders and subject matter experts. Risks associated with effectiveness of the programming was considered medium to high given the materiality of the programming, information needs of management, and program complexity (i.e. complex operating environment, varied nature of activities and longer timeframe for achieving outcomes). Therefore, the assessment of effectiveness would require greater effort, using administrative data review, interviews, document review and case studies. Risks associated with program efficiency/economy were considered to be low to medium, because of the availability of basic information on program costs, which could be supplemented by other types of information. The approach to evaluating efficiency and economy included document review, interviews and data analysis.

A review of approaches to evaluating community development planning and investment programming in other jurisdictions supported the use of a mixed methods research design to answer the key evaluation questions. The following methods were used for the study:

Administrative data review: An analysis of project data from ACOA’s QAccess database allowed for an in-depth understanding of the types activities funded within CM and CI programming. Beyond analyzing projects for their main activities, a separate analysis of the projects’ support to ACOA’s Strategy for Tourism Growth was conducted with the help of ACOA’s Tourism Atlantic group. All project coding exercises were validated through head office and regional program managers. Project data was supplemented by analysis of expenditure information from ACOA’s GX financial systems and project files where possible.
Document and literature review: The evaluation team reviewed relevant internal documents including: background documentation on the sub-programs and Transfer Payment Programs (TPP); program-related presentations and planning documents. A review of external documents included literature on federal priorities and related policies and strategies as well as documentation related to similar activities offered by other jurisdictions. The evaluation team also examined scholarly and grey literature relevant to the program theory or operating context.
Interviews[XVIII] : A total of 71 key informants were interviewed, many within the scope of specific case studies and others on the programming in general. Interviewees included a cross section of internal ACOA CD management (18 people took part in 6 group interviews), account managers (13), clients or beneficiaries (12), as well as other external stakeholders such as key funding partners, subject matter experts or other partner organizations (28). EAC and working group members provided advice on provincial government stakeholders to be interviewed.
Client Survey[XIX]: To reduce client burden it was determined that clients would only be surveyed about one project, the one that had taken place earliest within our timeframe to enable capturing outcomes. Two hundred twenty eight clients completed the survey to report on achievement of expected outcomes as well as the need for the programming, incrementality, and barriers and facilitators to success. For planning projects, of 51 participants invited, 41 (77 per cent) responded to the survey, 32 (64 per cent) of which had completed their project and contributed to the measurement of outcomes achievement. For investment projects, of 401 participants invited, 187 (47 per cent) responded to the survey, 154 (33 per cent) of which had completed their projects and contributed to the measurement of outcomes achievement. Survey findings were weighted to balance regional representation of clients. For instance projects from some smaller regions such as P.E.I and C.B. were slightly under- represented in the survey so the weight of responses from those regions was slightly enhanced to ensure regional representation in the survey was reflective of the regional distribution of project in the database..
Case studies: The evaluation team completed 13 case studies to more clearly understand ACOA’s contributions to the achievement of long term outcomes and impacts. These were grounded in a firm understanding of the context of ACOA’s interventions, and in a conceptual/analytical framework of the impact of ACOA’s CD programming. [XX] Case studies were selected to represent different types of projects. For each project type, a “best practice” case was studied to gain insights on how longer term outcomes are achieved, and a “lessons learned” initiative was studied to gain insights on key challenges and barriers to success. Case studies were undertaken early in the evaluation process to allow knowledge gained to contribute to the design and implementation of the other methods. Table B-1 details the types of case studies that were conducted as part of the evaluation.

Table B-1: Profile of Case Studies Conducted

Project Type	Types of Case		Region
Project Type	Achieved Outcomes	Lessons Learned
* symbol represents a follow up case study to the previous evaluation
Recreation/Cultural Infrastructure	Yes*	Yes	NS /NB
Basic Infrastructure	Yes	Yes	NS/ PEI
Marketing Industry	Yes	Yes	HO/NL
Planning Industry	Yes*	Yes	NB/NL
Tourism Attraction	Yes*	Yes	CB/PEI
Business Skills Development	Yes*		NL
Regional Development Planning and Mobilization - Indigenous	Yes		CB

*These were case studies that had been conducted as part of the previous program evaluation in 2009. A follow-up was conducted to capture longer-term impacts.

The evaluation questions and judgment criteria by which outcomes would be assessed are presented below in Table B-2. These also influenced the design of the evaluation.

Table B-2: Evaluation Questions, Judgment Criteria and Methods

When judgment criteria in the table below refer to the results of previous evaluations as a benchmark for assessing current program success, it should be noted that in some cases, baseline information will not be available. In those cases, this evaluation will be used to gather baseline data for future evaluations.

Evaluation Questions	Evaluation Judgment Criteria	Method
Evaluation Questions	Evaluation Judgment Criteria	Document Review	Interviews / Focus Groups	Surveys	Administrative Data	Case Studies
Relevance
Issue 1: Continued Need for the Programming
1.1. To what extent do the CM/ CI PSAs continue to address a demonstrable need?	The programming related needs are still present to at least the same degree as they were five years ago.	x	x			x
1.2 To what extent are the CM/ CI PSAs responsive to existing and emerging needs of Canadians? What other mechanisms exist to address these needs?	Community mobilization planning and investments respond to emerging needs. ACOA is aware of changing economic development needs and adjustments to programming are made to meet those needs. Roles and responsibilities of other service providers do not duplicate those of ACOA (extent of duplication, overlap or complementarity).	x	x			x
Issue 2: Alignment with Government Priorities
2.1 To what extent are the CM/ CI PSAs aligned with federal government priorities and expectations, and to ACOA’s strategic outcome?	There is logical alignment between the programming, federal government priorities (including evolving priorities) and ACOA’s strategic outcome. The alignment is recognized and/or made explicit.	x	x
Issue 3: Alignment with Federal Roles and Responsibilities
3.1 To what extent do the CM/ CI PSAs align with federal roles and responsibilities?	ACOA’s roles and responsibilities in this area of programming are aligned with the ACOA Act. Similar federal organizations in other jurisdictions administer such programming.	x	x
Performance
Issue 4: Effectiveness - the extent to which CM/ CI objectives have been achieved within the context of expected results and outcomes. [XXI]
4.1 How and to what extent are the CM/ CI PSAs achieving expected outcomes?		x	x	x	x	x
4.1.1 Incrementality: What impact would the absence of the CM/ CI programming have on projects/initiatives?	The impact of the absence of the programming is similar or greater to that observed in the previous evaluation, as reflected by: ACOA clients’ opinion of whether the project would have proceeded without funding ACOA’s influence on the involvement of funding partners			x
4.1.2 How and to what extent have CM/ CI PSAs contributed to immediate outcomes: Planning The ability/capacity to identify economic opportunities Leadership skills relating to economic development planning Improved collaboration between stakeholders in economic development planning Regional or sector specific economic development plans and strategies Investment Development of critical economic development infrastructure, including non- physical infrastructure Increased ability of communities to respond to opportunities and economic crisis Partnerships/collaboration for economic development Response to economic opportunities or crises.	There is sufficient evidence to argue the contribution of CM and CI programming to the achievement of immediate outcomes. The contribution of planning and investment activities to immediate outcomes is similar or greater to that observed in the previous evaluation [XXII] as reflected by: Planning 50 per cent projects or more will result in increased community or organization capacity for strategic planning relating to economic development; 50 per cent projects or more will result in increased leadership for strategic planning relating to economic development; 60 per cent or more projects will result in improved collaboration relating to economic development within the community or other organizations (stakeholders); 90 per cent of strategic plans have been finalized Plans meet “quality” criteria that is more likely to lead to economic development outcomes (i.e. prioritization of activities and partners that are making resource commitments, proposed initiatives build on assets for economic development, etc.); and/or Strategic plans identify opportunities that can lead to economic impacts such as employment/business creation, sustainability or growth. Investment The average number of partnerships/collaborators on projects is 2.7 or greater; All projects identify collaborators; 50 per cent projects have resulted in increased ability of communities to respond to subsequent opportunities and economic crisis; Internal tools and mechanisms allow for strategic decision making in the selection of projects that result in critical economic infrastructure (or non-physical infrastructure) [XXIII] being developed; and/or Projects resulted in a response to an economic opportunity or crisis. The Agency activity targets set as part of performance measurement for CM and CI have been achieved (e.g. # of projects, # of plans, and # of partnerships).	x	x	x	x	x
4.1.3 How and to what extent have the CM and CI PSA activities and immediate outcomes contributed to the achievement of intermediate outcomes: Planning The implementation of plans and strategies that have led to economic impacts. Investments Economic activities (within businesses, communities and industries) The creation of economic opportunities that have led to economic outcomes in the community.	There is sufficient evidence to argue the contribution of CM and CI programming to the achievement of intermediate outcomes. The contribution of planning and investment activities to intermediate outcomes is reflected by: Note: Current benchmarks for the following criteria are not available. Planning Plans contain initiatives that have been implemented and have increased capacity for economic development, as reflected by: Jobs created or maintained Reduced out migration Creation or survival of businesses Increased sales and revenues for businesses in the region Attraction of investment Increased exports Developed new markets Diversified exports Product development Business skills developed Tourism visitation influenced by planning activities. Investments Investment activities yield impacts that increase capacity for economic development, such as: Jobs created or maintained Reduced out migration Creation or survival of businesses Increased sales and revenues for businesses in the region Attraction of investment Increased exports Developed new markets Diversified exports Product development Business skills developed Tourism visitation Note: Case studies will play an important role in understanding the type of intermediate outcomes and impacts which can be expected, and the conditions or processes involved.	x	x	x		x
4.1.4 What are the barriers to achieving CM and CI immediate and intermediate outcomes and to what extent are these being mitigated?	Evidence of factors that are impeding success of the CM and CI PSAs are known, and mitigation strategies are implemented where applicable.	x	x	x		x
4.1.5 What lessons have been learned in the implementation of CM and CI programming? How can these lessons contribute to future programming?	Not applicable	x	x	x		x
4.1.5 What unintended outcomes have been achieved?	Not applicable		x	x		x
4.1.6 To what extent are the CM and CI PSAs performance measurement and reporting structures effective in reporting on the achievement of outcomes? How is the performance information used by ACOA-funded organizations and by ACOA?	Evidence that CM and CI PSA performance measurement is adequate and effective in reporting on the achievement of outcomes. Performance measurement information is used by ACOA and ACOA-funded organizations to support decision-making. Performance information is available for use in the evaluation.	x	x		x	x
Issue 5: Efficiency and Economy - the extent to which CM and CI activities are undertaken in an affordable manner, taking into consideration the relationship between outputs and the resources to produce them; the extent to which resources allocated to the CM and CI PSAs are well-utilized, taking into consideration alternative delivery mechanisms.
5.1 In the context of the results being achieved, to what extent are the resources (e.g. FTEs, financial) allocated to CM and CI PSAs efficiently utilized?	ACOA has in place mechanisms to ensure that the most efficient and economical means are being used to administer the programming. For example: Budget Plans Forecasts Resource Re-allocations (manner in which decisions were made) Governance/Processes Timeliness of delivery Delivery costs across regions reasonably compare to one another within the context of factors affecting regional delivery.	x	x		x
5.2 Is there a more efficient and effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned?	The organization takes into consideration its own best practices in improving alternative approaches (efficiency). Program management has considered and continues to explore alternative modes of delivery outside the organization. Best practices were identified and implemented.	x	x

Evaluation limitations and strategies applied to mitigate these are presented in table B-3 below.

Table B-3: Evaluation Challenges and Mitigation Strategies

Challenge	Mitigation
Shifting program priorities: During the evaluation period, the programming adapted to enable the delivery of the EAP programming and deliver on emerging Agency priorities.	Contextualized and described changes over time. Sought information on the effect of the changes on program delivery.
Limited information on planning projects: With the closure of REDOs, it was not possible to include them as evaluation participants, which limited the representation of planning projects in the client survey.	Described of funding decision, used existing performance data on REDO activities and acknowledged limitation. With regards to the interpretation of survey findings, numbers rather than percentages were reported where appropriate so as to contextualize the interpretation, acknowledging the limitation.
Program complexity: CM and CI programming aims to influence community economic development outcomes, which is affected by a broader system of variables and a wide range of stakeholders, making the study of CED complex. Also, CI programming aims to influence outcomes through a wide variety of initiatives, adding complexity to the study of program relevance and performance.	CD program theory was further developed in consultation with experts and key program stakeholders to better understand and acknowledge the context within CED outcomes are achieved. Evaluation was designed to capture complexity.
Data availability/reliability: Information on funded activities, performance and operational activities was not clear or consistently available.	To better understand funded activities, the team conducted a detailed analysis of all projects with input and validation from programs. The evaluation included methods, particularly the client survey and case studies, to gather performance information. These data are however subject to the limitations inherent to the survey and case studies methods. These include possible self-selection bias and other respondent biases and reductions in data validity due to the need to recreate data on past performance. The evaluation team developed detailed regional profile documents with input from programs to understand operations.
Difficulties in reaching participants: Interviewees could not always be reached for interviews. Similarly, survey participants could sometimes not be reached due to issues with outdated client contact information and change in personnel.	Interviews: Identified alternative interviewees Surveys: sent reminders, telephone option, email from ACOA, extended cut-off date.

The evaluation featured the following strengths:

Strong evaluation design: The evaluation was guided by developed program theory and incorporated of multiple methods to enable the study of short and long term outcomes. The evaluation constructed detailed regional profiles to obtain a solid understanding of the programming and differences in delivery and priorities. The evaluation methods and analysis allowed for consideration of various project types. Case studies enabled the evaluation team to capture complexity of influencing factors.
Stakeholder engagement: Evaluation methods were developed with the EAC program stakeholders, subject matter experts and client representatives (i.e. survey) to ensure all important issues were captured. The evaluation team worked closely with a working group comprised of managers from each ACOA region throughout the evaluation for fact checking and validation of analyses.
Survey response rate: A response rate of 50 per cent. Data was weighted to provide a proportionate representation of regions. Type of project was considered in the analysis of survey data.

Appendix C: Community Economic Development Conceptual/Analytical Framework

(D)

Appendix D: Management Action Plan

DATE: December 13, 2013

PROJECT TITLE: Evaluation of ACOA Community Mobilization and Community Investment Sub-programs

RESPONSIBILITY CENTRE: Community Development Program Activity

RESPONSIBILITY CENTRE MANAGER: Marc Lemieux, A./Director General, Community Development

Recommendation 1: In order to ensure a clear corporate direction for programming while allowing for regional variability and flexibility, ACOA should build on its corporate knowledge, lessons learned and current best practices in CED to formulate a Community Investment framework. The framework should outline the Agency’s approach to planning and investment programming (including main project types and expected outcomes), while acknowledging regional context, plans, priorities and best practices. It should also describe the Agency’s approach to engage, coordinate and collaborate both internally and externally (partners, subject matter experts) in the achievement of programming outcomes.

Management Response: Management agrees with the recommendation. Many of the elements of the framework suggested in the recommendation already exist in the Agency’s practices which will be built upon to develop the framework.

Planned Actions: The Framework will describe how the Agency will focus its efforts in the development of strategic investment plans as well as in its approach to support investments in community projects that contribute to the outcomes of community investment. In addition, the framework will consider the need for regional flexibility, outline best practices and highlight the Agency’s approach for engagement with key stakeholders.

Responsibility: Head Office Director General of Community Development and Directors General Programs/Operations Committee

Target Date: Fall 2014

Recommendation 2: In support of results-based management, decision-making, and the strategic coordination of initiatives that cut across programming areas (e.g. business skills development, tourism, trade, innovation), ACOA should ensure that quality and timely information on activities (i.e. main project types), and outcomes are systematically collected on file and in electronic systems.

Management Response: Management agrees with the recommendation. The performance information will support decision-making and capture learnings to enhance performance.

Responsibility: Head Office Director General of Community Development and Directors General Programs/Operations Committee

Target Date: March 31, 2014

Planned Actions: For projects that are coded under Community Investment, project types and their relationship to outcomes will be defined within the existing structure of the Agency’s Access system. The Community Investment Framework will make use of the information to support decision-making and facilitate linkages of projects to expected outcomes. The existing Results Tracking System found in the Agency’s Q Access system will be used to track project outcomes. Training and guidance will be provided to ACOA Program Officers on the project types, outcomes and quality of data to be inputted in the system. Monitoring on the quality of data entered in the system will be done on a regular basis.

Recommendation 3: In an effort to increase efficiency and effectiveness of programming and mitigate risks of not achieving expected outcomes, ACOA should build on current efforts to develop and systematically implement a standardized process/tool to assess project viability. The process/tool should assess the likelihood of achieving outcomes, as well as the human, financial, and infrastructure capacity of the proponent organization or community to conduct the work outlined in the proposal and to sustain the achievement of outcomes and impacts following project completion.

Management Response: Management agrees with the recommendation and will build on the current practices and tools already available.

Planned Actions: Develop and implement a risk assessment tool for non-commercial projects that will be used during the project assessment and documented within the existing Project Summary Form. Training and guidance will be provided to ACOA Program Officers on risk management for non- commercial projects.

Responsibility: Head Office Director General of Community Development and Directors General Programs/Operations Committee

Target: Fall 2014

Appendix E: Organizations Supporting CED Planning and Investment in Atlantic Canada

Organization	Programming Characteristics
ACOA	Focus on CED outcomes for Atlantic Canada Provides non-repayable funding to communities and industry sectors for: construction/infrastructure; equipment; marketing/attracting investment; skills development; events; tourism operations; immigration/recruitment of skilled labour; planning and studies Manages Infrastructure Canada programming (See Other Federal Departments)
ECBC	Delivers CM and CI programming on Cape Breton Island as well as other funding to non-commercial organizations for planning and investment activities Uses discretionary funding to support economic development initiatives; sometimes this is used to co-fund projects funded through CM and CI.
Other Federal Departments	Canadian Heritage (CH): Provides non-repayable funding to non-profit organizations for projects linked to Canadian history, culture, linguistic and geographic diversity such as festival; events; and planning studies for renovations and construction. Infrastructure Canada (IC): On behalf of IC, ACOA provides non-repayable contributions to Provincial governments, who support infrastructure relating to water and waste, municipal environmental energy improvements; transportation; tourism; and planning studies. Decisions are not based on economic development impacts Aboriginal Affairs and Northern Development Canada (AANDC): Provides non- repayable contributions for economic development organizations/services, training, capacity building, investment promotion, and research. Also provides non-repayable funds for core municipal infrastructure such as water and waste, not with an economic development lens.
Provincial Governments	Departments with economic development mandates NB-Regional Development Corporation; NL-Innovation, Business & Rural Development; NS-Economic and Rural Development and Tourism; PEI-Fisheries, Aquaculture, and Rural Development Non-repayable funds for planning and implementing economic development initiatives, including infrastructures relating to recreation, municipal services, events, etc. Types of support vary by region. Often co-fund CI projects with ACOA; provide less funding than ACOA generally, varies by project type and region Other Sector-Specific Provincial Departments Departments relating to tourism, culture, aquaculture and fisheries, natural resources, agriculture. Funded projects align with the departments’ key sectors and are typically represent smaller funding envelopes. Less frequent co-funders of ACOA CI projects.
Municipalities	Provide mainly funding for the development and maintenance of municipal infrastructures relating to transportation, tourism and recreation, along with events and local economic development planning (larger municipalities often have dedicated resources). Capital budgets based on tax base and municipality needs and priorities. Investments vary by type of projects and capacity to leverage funds from other partners.
CBDCs	Locally operated corporations that offer repayable loans and business support services, mostly to loan clients.
Other	UBDCs in NS are located in rural areas and provide advice and knowledge for businesses Provincially funded business support programs in NL target specific groups

Appendix F: Approved ACOA CM and CI Projects by Type and Sub- type, 2008-2009 to 2011- 2012

Planning or Investment	Project Type and Project Sub-Type	Projects		ACOA Approved
Planning or Investment	Project Type and Project Sub-Type	Number	per cent	$M	per cent
Investment	Construction/infrastructure	291	26.8	112.3	49.4
	community building Construction of community buildings that support the economic development of a region or community, usually rural. These include large multi-purpose centres.	34	3.1	23.3	10.3
	downtown/waterfront enhancement Improving downtown public spaces, including façade, streetscape and waterfront.	21	1.9	5.9	2.6
	enabling infrastructure Infrastructure needed for business or industry development /sustainability; including infrastructure for electricity; air transportation; sewer drainage; broadband; and transportation.	16	1.5	8.3	3.7
	industry sector Construction in support of key industries including cruise; mining; tourism; aerospace and defence; agriculture/agri-food; arts and crafts; forestry; IT; knowledge; manufacturing; meetings and conventions; and renewable/alternative energy.	38	3.5	22.0	9.7
	recreational/cultural Recreational sites that are related to tourism including nature trail/park; marina; sports centre/outdoor facility; and performing/visual arts facility.	77	7.1	16.7	7.4
	pre-commercial/industrial space Pre-development of commercial space to enable business; including farmer’s market, industrial centre, retail space/restaurant, incubation centre, and business offices.	24	2.2	9.8	4.3
	cultural/heritage Concert/event venue, heritage site/museums, and performance visual arts facilities.	69	6.4	21.0	9.1
	training/R&D infrastructure Industry training or R&D; including marine, agriculture/agri-food, sustainable/renewable energy, oil and gas, post-secondary.	11	1.0	5.3	2.3
Investment	Acquire equipment	73	6.7	14.3	6.3
	industry sector Support for industries including entertainment, transportation, fishery, manufacturing, mining, meetings and conventions, agriculture/agri-food, aquaculture, boat building, cruise, environment, and waste management.	22	2.0	5.3	2.4
	recreational/cultural Support for recreational/cultural events/activities/sites; including performing/visual arts facility, broadcasting facility, festival infrastructure, multi-purpose community centres, concert venue, museum/interpretive centre, and sports facility.	36	3.3	5.7	2.5
	training/R&D Support for the development of industry; including energy, manufacturing, mining, education, marine, and science and technology.	15	1.4	3.3	1.4
Investment	Event	166	15.3	13.2	5.8
	business development Awards/celebrations, workshops	7	0.6	<0.1	<0.1
	industry sector Hosting events related to an industry (i.e. film, mining, agriculture industries); including festivals, conferences, awards, trade shows, conference attendance, and expos.	57	5.2	1.8	0.8
	policy/public administration Hosting or participating in conferences related to policy or public administration.	2	0.2	0.2	0.1
	recreational/cultural Hosting or participating in recreational or cultural events; including conferences, celebrations/commemorations, and festivals	54	5.0	6.3	2.8
	regional economic development Hosting events related to promoting regional CED; including conferences and tradeshows.	23	2.1	0.9	0.4
	sporting/athletic Hosting tournaments, festivals, and celebrations/commemorations.	23	2.1	3.9	1.7
Investment	Immigration/recruitment	9	0.8	1.8	0.8
	immigration Enhancing immigration to the region including francophone and general immigrants.	2	0.2	<0.1	<0.1
	recruitment Recruiting labour: skilled workers, highly skilled workers, and international students.	7	0.6	1.7	0.8
Investment	Event Cultural or sporting events with tourism implications; including celebrations, festivals, and sporting events.	23	2.1	0.9	0.4
	industry sector Promoting specific industries including: arts and crafts, meetings and conventions, fisheries, film, agricultural/agri-food, oil and gas/marine, aerospace and defence, aquaculture, boat building, music, wine, and tourism.	15	1.4	11.2	5.0
	regional economic development Promoting a region for economic investments	3	0.3	0.2	0.1
	services Building awareness for services including offshore safety, and training and employment.	3	0.3	0.1	<0.1
	tourism attraction Attracting visitors to tourist areas or places; including region, golf experience, historic site, park/ecological reserve, museum/ interpretative centre, performing/visual arts, culinary experience, genealogy tourism, and nature trails.	74	6.8	13.6	6.0
	event Cultural or sporting events with tourism implications; including celebrations, festivals, and sporting events.	23	2.1	0.9	0.4
Investment	Skills development	76	7.0	10.4	4.6
	general SMEs Developing management, organizational and entrepreneurial development skills; including management skills/productivity, IT, IT management skills capacity, entrepreneurship, HR, innovation and commercialization, innovation and product development, investment readiness, land management, and supplier development.	29	2.7	4.9	2.2
	industry sector Training related to specific industries including technical attendance, technical develop/deliver, management skills/productivity, HR, SROI analysis, market readiness, tourism, and productivity.	43	4.0	4.8	2.1
	regional economic development Skills development related to community economic development.	4	0.4	0.6	0.3
Investment	Tourism attraction operation	11	1.0	1.6	0.7
	historic site Operational funding for historic sites.	5	0.5	0.8	0.4
	performing arts Operational funding for performing arts.	5	0.5	0.7	0.3
	museum/interpretive centre Operational funding for museum/interpretive centres.	1	0.1	0.1	0.1
Investment	Research centre operation	5	0.5	4.6	2.0
Investment	industry sector Operation funding for industry sectors; including fisheries, mining, and ocean technology	5	0.5	4.6	2.0
Planning	Planning	338	31.1	42.6	18.8
	industry sector Planning for the development of an industry overall; including tourism, oil and gas, mining, forestry, agriculture/agri-food, arts and crafts, fishery, music, transportation, aerospace and defence, forestry, harness racing, IT, knowledge, manufacturing, music, performing arts, renewable energy, and wine.	69	6.4	7.4	3.3
	project specific Planning for a particular initiative; including recreational/cultural infrastructure, recreational/cultural attractions, recreational/cultural events, port/harbour infrastructure, enabling infrastructure, downtown/waterfront infrastructure, business development, community buildings, academic, and regional tourism signage.	116	10.7	6.2	2.7
	regional development Planning for community economic development including rural, urban, indigenous, francophone, northern, and black business.	65	6.0	9.2	4.1
TOTAL	1087	100	226.96	100

Appendix G: CM and CI Projects by Main Types and Regional Office, 2008-09 to 2011- 2012

Project Type	NS
	No. Projects	Approved Funding
	No. Projects	$(M)	per cent
Construction/ Infrastructure	81	24.7	52.4
Acquire equipment	7	0.8	1.7
Event	8	3.4	7.0
Immigration/ recruitment	4	0.5	1.1
Marketing	15	1.8	3.9
Skills development	4	3.4	7.3
Tourism site operation
Research Centre Operation
Planning	85	12.5	26.5
Total	204	47.1	100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type	CB
	No. Projects	Approved Funding
	No. Projects	$(M)	per cent
Construction/ Infrastructure	47	16.8	55.4
Acquire equipment	23	3.1	10.3
Event	19	2.0	6.6
Immigration/ recruitment		��
Marketing	19	3.9	12.8
Skills development	6	0.7	2.2
Tourism site operation	1	<0.1	0.1
Research Centre Operation	1	0.5	1.6
Planning	42	3.3	11.0
Total	158	30.4	100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type	NL
	No. Projects	Approved Funding
	No. Projects	$(M)	per cent
Construction/ Infrastructure	77	29.8	46.3
Acquire equipment	22	4.1	6.4
Event	88	3.5	5.5
Immigration/ recruitment	1	<0.1	<0.1
Marketing	44	3.7	5.7
Skills development	27	3.5	5.4
Tourism site operation	5	0.9	1.4
Research Centre Operation	3	2.5	3.9
Planning	118	16.3	25.3
Total	385	64.4	100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type	PEI
	No. Projects	Approved Funding
	No. Projects	$(M)	per cent
Construction/ Infrastructure	56	12.5	29.8
Acquire equipment	12	1.3	3.1
Event	30	2.7	6.3
Immigration/ recruitment
Marketing	34	16.1	38.4
Skills development	5	0.9	2.1
Tourism site operation	4	0.6	1.5
Research Centre Operation
Planning	58	7.9	18.9
Total	199	42.0	100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type	HO
	No. Projects	Approved Funding
	No. Projects	$(M)	per cent
Construction/ Infrastructure
Acquire equipment
Event	4	0.1	6.7
Immigration/ recruitment	1	<0.1	1.3
Marketing	1	0.3	12
Skills development	5	0.6	32.7
Tourism site operation
Research Centre Operation
Planning	9	0.9	47.3
Total	20	1.9	100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type	NB
	No. Projects	Approved Funding
	No. Projects	$(M)	per cent
Construction/ Infrastructure	30	28.5	69.1
Acquire equipment	9	5.0	12.1
Event	17	1.6	4.0
Immigration/ recruitment	3	1.2	2.9
Marketing	5	0.3	0.6
Skills development	29	1.3	3.2
Tourism site operation	1	0.1	0.2
Research Centre Operation	1	1.6	3.9
Planning	26	1.7	4.2
Total	121	41.2	100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Appendix H: Annual Salary Costs per Region

Table H-1: Community Mobilization and Community Investment Salary per Region by Fiscal Year

Region	Salary ($)
Region	2008-09	2009-10	2010-11	2011-12	Total ($)
N.L.	1,496,248	1,829,430	1,853,406	1,866,929	7,046,013
N.S.	1,156,315	1,445,546	1,218,942	1,063,134	4,883,937
N.B.	1,934,500	1,683,814	1,154,064	1,325,458	6,097,837
P.E.I.	204,990	545,207	656,953	599,752	2,006,902
H.O.	299,824	810,133	706,646	672,755	2,489,358
C.B.	473,602	582,154	585,203	519,913	2,160,872
Total	5,565,479	6,896,284	6,175,214	6,047,941	24,684,919

Source: GX Financial Data, August 2012

Appendix I: End Notes

[1] Government Organization Act, Atlantic Canada, 1987, Part 1, R.S., c G-5-7.

[2] TB Policy Suite on Evaluation: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx? section=text&id=15024

[3] Patton, M.Q. (2008) Utilization-Focused Evaluation, 4^th Edition

[4] Europaid Cooperation office, extracted May 2012.

[5] ACOA. Innovative Communities Fund Terms and Conditions. 2011.

[6] ACOA. Business Development Program Terms and Conditions.

[7] ACOA. Evaluation of Community Investment and Community Development Resources Program Sub-Activities. 2009.

[8] ACOA Departmental Performance Report 2010-2011

[9] Chen, H.T. (1990). Theory-driven evaluations. Sage Publications.

[10] Ibid.

[11] TBS (2012). Theory-Based Approaches to Evaluation: Concepts and Practices. http://www.tbs-sct.gc.ca/cee/tbae-aeat/tbae-aeat- eng.pdf

[12] ACOA Departmental Performance Report 2011 - Atlantic Canada’s 2010-2011 Economic Overview and Provincial Economic Profiles.

[13] ACOA, Community Development Impact Model, p.8.

[14] Vasseur, L. & Catto, N. R. (2008). Atlantic Canada; in From Impacts to adaptation: Canada in a changing climate 2007, edited by Lemmen, D. S., Warren, F. J., Lacroix, J. & Bush, E. Government of Canada, Ottawa, ON, p. 119-170.

[15] ACOA, Community Economic Development Impact Framework, (March, 2012).

[16] ACOA, Community Investment and Community Development Resources Evaluation, (Dec. 2009).

[17] Royal Bank of Canada Provincial Outlook 2013

[18] Community Investment and Community Development Resources Evaluation, (Dec. 2009).

[19] APEC: Meeting the skills challenge, (2012).

[20] APEC: Meeting the skills challenge, (2012).

[21] Bourgeois, Surette & Thériault, The knowledge economy in Atlantic Canada. Centre for Innovation and Productivity, (2011).

[22] APEC: Meeting the skills challenge, (2012).

[23] Statistics Canada Labour Force Survey

[24] Roger Martin and James B. Milway, Institute for Competitiveness and Prosperity, Strengthening Management for Prosperity (May 2007), page 18.

[25] Penelope M. Rowe, The Non-Profit and Voluntary Sector in Atlantic Canada: Regional Highlights of the National Survey on Non-Profit and Voluntary Organizations, (2006).

[26]Richard Matoon, Paper on the role of infrastructure for economic development, Economic Conference, Ottawa Canada (2004).

[27] Statistics Canada, Age of Public Infrastructure: A Provincial Perspective written by Michele Gagnon, Valerie Gaudreault and Donald Overton, (2008).

[28] Penelope M. Rowe, The Non-Profit and Voluntary Sector in Atlantic Canada: Regional Highlights of the National Survey on Non-Profit and Voluntary Organizations, (2006).

[29]Conference Board of Canada, Canada Enters a Period of Fiscal Restraint, ( March 31, 2011): http://www.conferenceboard.ca/press/newsrelease/11-03- 31/canada_enters_a_period_of_government_fiscal_restraint.aspx

[30] Goss Gilroy Inc. Summative Evaluation of the Economic Development Initiative, under the 2008-2009 Strategy for Official Languages: Roadmap for Canada’s Linguistic Duality. (July, 2012)

[31] Mount Allison University, Baseline Data for Aboriginal Economic Development: An Informed Approach for Measuring Progress and Success, (Nov. 2010).

[32] ACOA, Program Alignment Architecture, 2011-2012.

[33] Treasury Board Secretariat of Canada, Whole-of-government framework, I Economic Affairs:

[34] 2008-2011, 2012 Federal Budget Speeches

[35] Speech From the Throne 2008, 2009; Budget 2012

[36] Speech From the Throne 2008

[37] Speech From the Throne 2010-2011 and Budget 2012

[38] Speech From the Throne 2010

[39] ACOA Act, (R.S.C., 1985, c. 41 (4th Supp.) http://laws- lois.justice.gc.ca/eng/acts/A-13.7/page-1.html

[40] Idem.

[41] ACOA DPRs, 2008-9 to 20011-12:

[42] Goss Gilroy. Summative Evaluation of the Economic Development Initiative Under A Roadmap for Canada’s Linguistic Duality 2008-2013. 2012

[43] ACOA. Évaluation du programme sur l’immigration francophone (Phase 1). 2012

Footnotes

[I]In order to properly contextualize findings, some references have been made to activities occurring outside the scope of this evaluation.

[II] Due to challenges associated with segregating costs of delivering initiatives associated with EAP and Roadmap for Linguistic Duality, the evaluation of efficiency and economy included these as part of all programming delivered under CI.

[III] Two case studies were completed by Performance Management Network, and the client survey was administered by Goss Gilroy Inc.

[IV] The overall number of projects was 1,087, and these were conducted by 556 client organizations. After removing REDOs, organizations whose projects had just begun (due to their inability to report on outcomes), and organizations whose contact information was invalid, the number of individual clients surveyed was 465.

[V] These impacts were identified in consultation with program managers during the evaluation planning phase and development of the CED Conceptual/Analytical Framework

[VI]Due to the complexity of the programming, the evaluation unit expanded upon the logic models to develop a broader program theory, in consultation with program and subject-matter experts. The CED Conceptual/Analytical Framework outlines the relationship between ACOA CD programming activities and their interaction.

[VII] Economic challenges abroad such as failure of the US economy to make gains, the European recession and slowdown in Chinese economic growth, fostered a challenging environment for economic recovery in Canada overall.

[VIII] ACOA. Evaluation of the CI, CDR, OLMC and Aboriginal Communities Program Sub-Activities, 2009

[IX] It should be noted that outcome measurement for initiatives that resulted from planning would have likely benefited from a longer measurement time frame.

[X] As some regions adopted the practice of distributing new funding through amendments to REDO projects, approval amounts are lower than actual expenditures. Actual expenditures for REDO operational activities were $29.6 million over the four-year period.

[XI]These results represent the number and percentage of completed projects for which proponents reported the achievement of outcomes to either some extent or a great extent.

[XII] These results represent the number and percentage of completed projects for which proponents reported the achievement of outcomes to either some extent or a great extent.

[XIV] This time frame was most relevant for studying linkages to ACOA’s Strategy for Tourism Growth, as it was implemented by early 2009.

[XV] The analysis of projects in support of tourism was supported by ACOA’s Tourism Atlantic group and further validated by regional CD managers and directors.

[XVI] The survey asked proponents to report the number of other organizations involved in their projects.

[XVII]DPR targets for the number of collaborators per project were largely met and ranged from 650 collaborators in 2009-2010 to 324 in 2011-2012.

[XVIII] Due to ACOA’s decision to end core operational funding under the current REDO model effective May 2013, these organizations will be excluded from interviews.

[XIX] Due to ACOA’s decision to withdraw end core operational funding under the current REDO model effective May 2013, these organizations will be excluded from the client survey.

[XX] ACOA (2012). Community Economic Development Conceptual/Analytical Framework. Draft Project Report.

[XXI] Generally, the wording of expected outcomes is derived from the logic models. However, as mentioned in Section 2.1, it was found in consultations that the original logic models for CM and CI did not fully capture desired outcomes and changes were suggested. Wording of expected outcomes has been modified to reflect the consultation findings.

[XXII] The targets (%) were identified based on previous evaluation findings.

[XXIII]In the 2009 evaluation of CI/CDR, ACOA adopted an OECD definition of critical infrastructure for economic development: Canada’s critical infrastructure consists of those physical and information technology facilities, networks, services and assets which if disputed or destroyed, would have a serious impact on the economic well-being of Atlantic Canadians.

Page details

Report a problem on this page

Date modified:: 2021-11-23

Evaluation of the Atlantic Canada Opportunities Agency Community Mobilization and Community Investment Sub-Programs

List of Acronyms

Table of Contents

List of Figures

List of Tables

Acknowledgements

Executive Summary

1. Introduction

Evaluation Overview

1.2 Evaluation Design and Methodology

1.3 Evaluation Limitations, Mitigating Measures and Strengths

2. Profile of the Community Mobilization and Community Investment Sub- programs

2.1 Context

2.2 Special Initiatives

2.3 Program Theory

2.3.1 Program Logic Model

2.3.2 Other Influential Factors

2.4 Program Accountability and Governance

2.5 Expenditure Profile

2.5.1 Proportion of Overall Agency Expenditures

2.5.2 CM and CI Expenditures

2.6 Project Profile

3. Findings: Relevance

3.1 Continued Need for Programming

3.1.1 Programming-Related Needs

3.1.2 Awareness of and Responsiveness to Changing Needs

3.1.3 Extent of Overlap, Duplication and Complementarity with Other Programming

3.2 Alignment with ACOA and Government-wide Priorities and Strategies

3.3 Alignment with Federal Roles and Responsibilities

4. Findings: Performance – Effectiveness

4.1 Incrementality

4.2 Achievement of Expected Results

4.2.1 Planning Projects

4.2.2 Investment Projects

4.3 Unintended Outcomes of Planning and Investment Programming

4.4 Roadmap for Linguistic Duality Initiatives: Achievement of Outcomes

4.5 Overall Achievement of CED impacts

4.6 Barriers to the Achievement of Planning and Investment Outcomes

4.7 Lessons Learned and Best Practices in Delivery

4.8 Adequacy of Performance Measurement

5. Findings: Performance – Efficiency and Economy

5.1 Efficient Utilisation of Resources

5.1.1. Delivery Costs

5.1.2 Mechanisms that Support Efficient and Economical Delivery

5.2 Alternative Modes of Delivery for Increasing Efficiency and Economy

6.0 Conclusions and Recommendations

Appendix A: Program Alignment Architecture Chart

Appendix B: Evaluation Design and Methods

Appendix C: Community Economic Development Conceptual/Analytical Framework

Appendix D: Management Action Plan

Appendix E: Organizations Supporting CED Planning and Investment in Atlantic Canada

Appendix F: Approved ACOA CM and CI Projects by Type and Sub- type, 2008-2009 to 2011- 2012

Appendix G: CM and CI Projects by Main Types and Regional Office, 2008-09 to 2011- 2012

Appendix H: Annual Salary Costs per Region

Appendix I: End Notes

Footnotes

Page details