Evaluation of the Atlantic Canada Opportunities Agency community mobilization and community investment sub-programs

 

Evaluation Unit, Evaluation and Risk Directorate
Atlantic Canada Opportunities Agency
February 2014

List of acronyms

A&D - aerospace and defence

ACOA - Atlantic Canada Opportunities Agency

BDP - Business Development Program

CEAP - Canada’s Economic Action Plan

CED - community economic development

CAF - Community Adjustment Fund

CD - Community Development

CED-Q - Canada Economic Development for Quebec Regions

CI - Community Investment

CM - Community Mobilization

DG - director general

DPR - departmental performance report

EAP - Economic Action Plan

ECBC - Enterprise Cape Breton Corporation

ED - Enterprise Development

EAC - evaluation advisory committee

ExCom - Executive Committee

FedNor - Federal Economic Development Initiative for Northern Ontario

FTE - full-time equivalent

G&Cs - grants and contributions

HO - head office

ICF - Innovative Communities Fund

O&M - operations and maintenance

OLMC - official language minority community

PAA - program alignment architecture

PAC - Policy, Advocacy and Coordination

PMS - performance measurement strategy

REDO - regional economic development organization

RDA - regional development agency

RInC - Recreational Infrastructure Canada

SME - small and medium-sized enterprises

TB - Treasury Board

TBS - Treasury Board of Canada Secretariat

VP - vice-president

WD - Western Economic Diversification Canada

Table of contents

Acknowledgements

Executive summary

1.0 Introduction

1.1 Evaluation overview

1.2 Evaluation design and methodology

1.3 Evaluation limitations, mitigating measures and strengths

2.0 Profile of the community mobilization and community investment sub-programs

2.1 Context

2.2 Special initiatives

2.3 Program theory

2.3.1 Program logic model

2.3.2 Other influential factors

2.4 Program accountability and governance

2.5 Expenditure profile

2.5.1 Proportion of overall Agency expenditures

2.5.2 CM and CI expenditures

2.6 Project profile

3.0 Findings: relevance

3.1 Continued need for programming

3.1.1 Programming-related needs

3.1.2 Awareness of and responsiveness to changing needs

3.1.3 Extent of overlap, duplication and complementarity with other programming

3.2 Alignment with ACOA and government-wide priorities and strategies

3.3 Alignment with federal roles and responsibilities

4.0 Findings: performance – effectiveness

4.1 Incrementality

4.2 Achievement of expected results

4.2.1 Planning projects

4.2.2 Investment projects

4.3 Unintended outcomes of planning and investment programming

4.4 Roadmap for linguistic duality initiatives: achievement of outcomes

4.5 Overall achievement of CED impacts

4.6 Barriers to the achievement of planning and investment outcomes

4.7 Lessons learned and best practices in delivery

4.8 Adequacy of performance measurement

5.0 Findings: performance – efficiency and economy

5.1 Efficient utilisation of resources

5.1.1. Delivery costs

5.1.2 Mechanisms that support efficient and economical delivery

5.2 Alternative modes of delivery for increasing efficiency and Economy

6.0 Conclusions and recommendations

Appendix A: Program alignment architecture chart

Appendix B: Evaluation design and methods

Appendix C: Community economic development conceptual/analytical framework

Appendix D: Management action plan

Appendix E: Organizations supporting CED planning and investment in Atlantic Canada

Appendix F: Approved ACOA CM and CI projects by type and sub-type, 2008-2009 to 2011-2012

Appendix G: CM and CI projects by main types and regional office, 2008-09 to 2011-2012

Appendix H: Annual salary costs per region

Appendix I: End notes

Footnotes

List of figures

Figure 1: Incrementality of CM and CI programming

Figure 2: Community mobilization and community investment expenditures by region (2008-2009 to 2011-2012)

Figure 3: Operational spending per region

Figure 4: Alignment between CM and CI evaluation conclusions and recommendations

List of tables

Table 1: Evaluation questions by core issue

Table 2: Community mobilization and community investment logic models

Table 3: Expenditures ($M) by program, 2008-2009 to 2011- 2012

Table 4: Annual expenditures for CM and CI, 2008-2009 to 2011- 2012

Table 5: Annual G&C expenditures for CM and CI ($M), 2008-09 to 2011- 12

Table 6: Planning and investment projects approved 2008-2009 to 2011- 2012

Table 7: Programming similar to CM and CI delivered by RDAs in other Canadian jurisdictions

Table 8: Types of planning projects and funding approved, 2008-09 to 2011- 12

Table 9: Types of activities funded by investment projects, 2008-2009 to 2011- 2012

Acknowledgements

This evaluation was undertaken to provide the Atlantic Canada Opportunities Agency (ACOA) management with systematic, neutral evidence on the relevance and performance of its Community Mobilization (CM) and Community Investment (CI) sub-programs. It is expected that the results of this study will be used to support the future direction of programming. The study was managed and completed by ACOA’s Evaluation Unit.

We would like to take this opportunity to thank the members of the evaluation advisory committee and the evaluation working group for their advice and support throughout this process. Their assistance helped to ensure the relevance and usefulness of this evaluation. Of note is the gracious contribution of the director general for community development, the regional program directors and managers, and three external members: Dr. Kelly Vodden, Memorial University of Newfoundland; Lise Ouellette, Community College of New Brunswick; and Daniel Fairbarn, Industry Canada.

We are also grateful to the many ACOA staff members, including staff from the Evaluation Unit, as well as ACOA clients and external key informants who provided their time and essential knowledge in support of this study.

Overall, these contributions were instrumental in building our understanding and correctly conveying the nature of the community development planning and investment programming at ACOA.

Courtney Amo Acting Director and Head of Evaluation, Evaluation and Risk Directorate Atlantic Canada Opportunities Agency

Executive summary

ACOA’s Community Mobilization (CM) and Community Investment (CI) sub-programs are part of the Agency’s Community Development (CD) program. Through its CD program, ACOA works with communities and entrepreneurs to stimulate economic growth, improve infrastructure and develop opportunities in local economies, with the desired outcome of creating dynamic and sustainable Atlantic communities.

In support of this outcome, the CM sub-program supports the mobilization of key stakeholders for the planning of economic development initiatives, while the CI sub-program invests in critical economic development infrastructure and other initiatives that support economic development outcomes.

CM and CI programming is delivered using a decentralized model, involving all ACOA regional and district offices working in close partnership with a variety of government and non-governmental stakeholders. Two main transfer payment programs are used: The Innovative Communities Fund (ICF) and the business support element of the Business Development Program (BDP). Overall, an average of approximately $103 million per year is expended through CM and CI programming, representing 28 per cent of ACOA’s total expenditures.

The purpose of this evaluation is to assess the relevance and performance of CM and CI programming and to fulfill Government of Canada accountability requirements. The evaluation covers the four-year period from fiscal year 2008-2009 to 2011-2012. Although CM and CI are separate sub-programs, they have been evaluated together to acknowledge the complementarity between economic development planning and investments and to address challenges in how the sub-programs are differentiated in administrative data. This is in keeping with the terms of reference for the study, which were approved by ACOA’s executive committee in June 2012.

The evaluation methodology included 13 case studies, 71 key informant interviews, a client survey with an overall response rate of 49 per cent, a document and literature review, and a detailed analysis and categorization of over 1,000 projects. Close to 300 individuals provided their knowledge and insight to this study. Given that operational funding to regional economic development organizations (REDOs) across Atlantic Canada was ceased as of May 2013, only existing data related to REDOs was considered in this evaluation. Evaluation findings are based on a high level of convergence of multiple lines of evidence and are deemed reliable and valid within the context of the study limitations.

Findings and conclusions

Relevance

The CM and CI sub-programs are relevant activities for ACOA and the federal government. The programming aligns with ACOA’s strategic outcome and the ACOA Act, and activities are consistent with the Agency’s roles and responsibilities and broader federal priorities.

There is a continued need for planning and investment programming as challenges to community economic development (CED) in Atlantic Canada exist to the same degree or are greater than reported in the previous evaluation. In particular, the evaluation reports a substantial increase (around 30 per cent) since the last evaluation in the number of clients who cite challenges related to the aging population and outmigration as key barriers to economic development.

In response to changing needs, the Agency has adjusted its programming. ACOA has endeavoured to invest in projects that generate greater economic impacts (including economic growth and job creation) and has discontinued operational funding to REDOs across Atlantic Canada in order to reduce duplication of services, to streamline delivery and to acknowledge the changing demographics of rural areas.

ACOA’s CM and CI programming is unique and complementary to CED programming offered by other organizations, and is similar to what is offered through other regional development agencies (RDAs). The programming’s distinctive traits include: its Atlantic-wide scope; the contributions of ACOA’s Policy, Advocacy and Coordination (PAC) function; internal capacity for proposal development and assessment; programming flexibility; and focus on CED goals.

A collaborative, coordinated and strategic approach among partners is critical to the achievement of CED outcomes. ACOA works with, and alongside, a variety of partners and stakeholders toward these outcomes. The dissolution of the REDO model brings about a necessary period of transition as ACOA and CED partners realign resources and clarify roles and responsibilities. During the transition period, there is a risk of capacity gaps for regional-level planning or other economic development activities.

Performance

Incrementality

CM and CI programming is incremental to the implementation of projects, to obtaining investments from other partners and to achieving results. Almost two thirds (64 per cent) of surveyed clients reported that their projects would likely not have proceeded at all without ACOA’s support; and of those projects that could have proceeded, all but 3 per cent report negative impacts, including smaller scope, time delays, poorer quality and difficulty securing other funding. Over two thirds (70 per cent) of surveyed clients indicated that other funding was secured as a result of ACOA’s decision to provide support. On average, clients report that 42 per cent of total project funding was secured from external sources as a result of ACOA. These results were echoed by key informants, who reported on the influence played by ACOA’s investment decision.

Achievement of expected outcomes

CM and CI programming is incremental to the implementation of projects, to obtaining investments from other partners and to achieving results. Almost two thirds (64 per cent) of surveyed clients reported that their projects would likely not have proceeded at all without ACOA’s support; and of those projects that could have proceeded, all but 3 per cent report negative impacts, including smaller scope, time delays, poorer quality and difficulty securing other funding. Over two thirds (70 per cent) of surveyed clients indicated that other funding was secured as a result of ACOA’s decision to provide support. On average, clients report that 42 per cent of total project funding was secured from external sources as a result of ACOA. These results were echoed by key informants, who reported on the influence played by ACOA’s investment decision.

Achievement of expected outcomes

Overall, the evaluation has found that the majority of both planning and investment projects achieved the immediate results expected of CM and CI programming.

There were four main types of planning projects over the evaluation period: 1) support for REDO operations, including core planning activities; 2) other regional economic development planning; 3) industry sector development planning; and 4) studies or plans related to specific projects or initiatives.

In general, planning projects resulted in partner mobilization, capacity building and the development of concrete plans, which supported increased CED skills and leadership, decision-making influence and concrete actions for economic development. Due to low survey representation of projects with completed plans and strategies, it is not possible to develop conclusions on the overall achievement of intermediate outcomes of planning activities. However, survey findings and case studies support the importance of thorough project planning for greater economic development impacts.

Over the evaluation period, the largest category of investment projects was related to construction of infrastructure (61 per cent of funding). Within this category, 55 per cent of funding supported community, recreational and cultural infrastructure. These projects and their expected outcomes are not well articulated in CI programming documents. In addition, project analysis revealed that a large proportion of investment funding was linked to tourism, either by supporting ACOA’s Growth Strategy for Tourism (35 per cent or $49.7 million), or by having other tourism implications (27 per cent or $38.3 million). The extent to which CI programming aims to target the tourism sector and its importance in relation to other types of projects is not clear in CI documentation.

Beyond the funding that supported tourism and community infrastructure for recreation and culture, a combined 35 per cent ($64.5 million) of funding supported: infrastructure and equipment purchases for industry/sector-related training, research and other activities; infrastructure that enables economic activity such as broadband, sewage/drainage and electrical upgrades; buildings that accommodate commercial space; and training to enhance business or industry-related skills.

Investment projects generally achieved immediate outcomes such as creation of infrastructure for economic development and increased capacity for responding to future opportunities. Intermediate outcomes included increased tourism visitation and sales and revenues for businesses, as well as the development of new products. There was evidence of modest job creation and maintenance as well as modest business creation and survival.

Overall, the evaluation provides evidence that the capacity and asset base of communities is strengthened as a result of planning and investment funding offered by ACOA and its partners. By strengthening their capacity and asset base, communities can better respond to economic opportunities and threats, increase their economic stability and sustainability; and improve the quality of life of residents. However, the evaluation cannot speak to the overall impact of this strengthened asset base on the economic viability and sustainability of communities.

Barriers to achieving results exist, and could be better mitigated through improved project viability/risk assessment processes and the adoption of promising practices tested in some regions. In addition, the extent of coordination of projects that cut across ACOA programming areas (e.g. business skills development, tourism, trade, innovation) was unclear, which may create inefficiencies in achieving Agency objectives relating to those areas. Limited information on activities, outputs and outcomes for individual projects and at the aggregate level presents challenges to this type of coordination and to the strategic, results-based management of CM and CI programming.

Overall, the CM and CI programming was delivered in a cost-effective manner. Internal costs were proportionate to the amount of funding delivered and compared reasonably between ACOA regions. ACOA support enabled projects to leverage substantial funds from other organizations. Leveraging of other funding has increased since the previous evaluation. Mechanisms that fostered efficient and economical delivery included enhanced planning processes, efforts to achieve greater economic impacts, collaboration among ACOA units, and coordination with other RDAs in the delivery of national initiatives. ACOA was efficient in delivering substantial EAP programming without additional dedicated resources; however, this resulted in fewer projects related to ACOA’s core CM and CI programming being delivered.

Recommendations

The evaluation team has issued three recommendations, which are summarized below and further detailed in the report. The recommendations are aimed at improving programming by balancing a consistent, targeted approach with required flexibility; improving the availability and access of information and knowledge to manage for results; and building on existing and new relationships, expertise and intelligence to narrow the gap between ACOA’s community economic development theory and practice, as implemented through the CM and CI sub-programs.

Recommendation 1

In order to ensure a clear corporate direction for programming while allowing for regional variability and flexibility, ACOA should build on its corporate knowledge, lessons learned and current best practices in CED to formulate a Community Investment framework. The framework should outline the Agency’s approach to planning and investment programming (including main project types and expected outcomes), while acknowledging regional context, plans, priorities and best practices. It should also describe the Agency’s approach to engage, coordinate and collaborate both internally and externally (i.e. with partners and subject matter experts) in the achievement of programming outcomes.

Recommendation 2

In support of results-based management, decision making, and the strategic coordination of initiatives that cut across programming areas (e.g. business skills development, tourism, trade, innovation), ACOA should ensure that quality and timely information on activities (i.e. main project types) and outcomes are systematically collected on file and in electronic systems.

Recommendation 3

In an effort to increase efficiency and effectiveness of programming and mitigate risks of not achieving expected outcomes, ACOA should build on current efforts to develop and systematically implement a standardized process/tool to assess project viability. The process/tool should assess the likelihood of achieving outcomes as well as the human, financial, and infrastructure capacity of the proponent organization or community to conduct the work outlined in the proposal and to sustain the achievement of outcomes and impacts following project completion.

Management has agreed with the evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the evaluation’s recommendations, can be found in Appendix D.

1. Introduction

This report presents the findings, conclusions and recommendations of the evaluation of ACOA’s Community Mobilization (CM) and Community Investment (CI) sub-programs. The programming was evaluated for relevance and performance in accordance with the Treasury Board (TB) Policy on Evaluation, the Agency’s approved evaluation plan, and the approved terms of reference for this study.

The CM and CI sub-programs support the achievement of ACOA’s Community Development (CD) program goal to foster dynamic and sustainable Atlantic communities and more broadly, the Agency’s mandate “to increase opportunity for economic development in Atlantic Canada” [1]. CM programming focuses on mobilizing stakeholders and planning for regional or community economic development initiatives. CI programming invests in critical economic development infrastructure and other initiatives that support economic development outcomes.

The evaluation examined a four-year period, from 2008-2009 to 2011-2012 [I] . The scope of the evaluation excludes measuring outcomes achievement for programming related to Canada’s Roadmap for Linguistic Duality and the Economic Action Program (EAP) as they were reported on separately. However, the evaluation considered best practices and lessons learned, particularly from the delivery of EAP programming, to inform future delivery of similar short-term initiatives. Also, these initiatives were considered as part of the programming delivered by CI staff in the evaluation’s assessment of CM and CI efficiency and economy (Section 5). [II]

Following the evaluation overview presented below, Section 2 of the report provides a profile of the programming. Sections 3 through 5 present the study’s findings, organized by broad evaluation question (relevance, performance-effectiveness, performance-efficiency and economy), and Section 6 presents the conclusions and recommendations resulting from the study.

Management has agreed with the evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the evaluation’s recommendations, can be found in Appendix D.

1.1 Evaluation overview

The purpose of this evaluation was twofold. First, to provide timely, credible and neutral information on the relevance and performance of CM and CI programming in order to support decision-making, continuous improvement and results-based management. Second, to fulfill accountability requirements set forth by the TB Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada [2].

The research was completed between June 2012 and June 2013. An extensive planning phase that examined all sub-programs within the CD program ran from August 2011 to June 2012. During this phase, it was determined that the CM and CI sub- programs should be evaluated together given the complementarity between economic development planning and investments, and inconsistencies in the coding of planning and investment projects to each sub-program within ACOA’s administrative database. To ensure that projects were considered appropriately in the evaluation, the evaluation team conducted a detailed review of all projects to identify the nature of the activities under the two broad categories of planning or investments. Therefore, this report refers to the overall set of projects considered in the evaluation as economic development “planning and investment programming.”

ACOA’s Evaluation Unit planned and conducted the evaluation with the assistance of consulting firms for two case studies and the client survey [III]. An evaluation advisory committee (EAC), with representatives from ACOA management and staff as well as external stakeholders with specialized knowledge in community economic development (CED), provided advice and guidance throughout the evaluation. A working group of managers from all ACOA regions and Enterprise Cape Breton Corporation (ECBC) provided access to information and other resources. The EAC and working group helped ensure clear understanding of the programming and supported the interpretation of findings, which added to the relevance and usefulness of the final report and its recommendations.

1.2 Evaluation design and methodology

The evaluation of the CM and CI sub-programs addresses five core issues that fall into two broad categories, relevance and performance, in accordance with the TB Policy on Evaluation. Table 1 identifies the specific evaluation questions for each core issue. These were developed during the planning phase that preceded the evaluation, when key stakeholders were able to raise possible issues for further study.

Table 1: Evaluation questions by core issue

Issue

Evaluation Question

Relevance

Issue 1: Continued Need for the Program

1.1

To what extent do the CM/ CI sub-programs continue to address a demonstrable need?

1.2

To what extent are the CM/ CI sub-programs responsive to the existing and emerging needs of Canadians? What other mechanisms exist to address these needs?

Issue 2: Alignment with Government Priorities

2.1

To what extent are the CM/ CI sub-programs aligned with federal government priorities and expectations, and to ACOA’s strategic outcome?

Issue 3: Alignment with Federal Roles and Responsibilities

3.1

To what extent do the CM/ CI sub-programs align with federal roles and responsibilities?

Performance

Issue 4: Effectiveness

4.1

How and to what extent are the CM/ CI sub- programs achieving expected results (immediate and intermediate outcomes)? What impact would the absence of the CM/ CI programming have on projects/initiatives? What are the barriers to the achievement of expected outcomes, and to what extent have these been mitigated? What lessons have been learned, and how can these contribute to future programming? What unintended outcomes have been achieved? To what extent are the CM/ CI sub- programs’ performance measurement and reporting structures effective in reporting on the achievement of outcomes? How is the information used by ACOA and ACOA-funded organizations?

Issue 5: Efficiency and Economy

5.1

In the context of the results being achieved, to what extent are the resources allocated to the CM and CI sub-programs efficiently utilized?

5.2

Is there a more efficient and effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned?

 

Judgment criteria, or “benchmarks” for successful achievement, were developed with advice from program management for each evaluation question [3] . The evaluation team used the judgment criteria to objectively and transparently assess the relevance and performance of the programming. The judgment criteria were considered in determining data needs, methodology and analysis. [4]

A risk-based approach was used to design the evaluation and methods, as required by the TB Policy on Evaluation. The level of effort associated with conducting the evaluation was calibrated to reflect risks associated with the relevance, effectiveness and efficiency/economy of the programming, as determined through a document review and stakeholder consultations. Considering evaluation issues, risk and judgment criteria, a mixed methods research design was chosen involving multiple lines of evidence gathered through the following methods:

See Appendix B for further detail on the evaluation design, calibration of effort, methods, and judgment criteria.

Note that the evaluation team did not collect new performance information from regional economic development organizations (REDOs) through the client survey or key informant interviews due to ACOA’s cessation of operational funding during the evaluation process (see section 2.1 for further information). However, existing performance measurement data related to the REDOs were considered in assessing outcome achievement.

1.3 Evaluation limitations, mitigating measures and strengths

The evaluation design and implementation are considered appropriate based on the intended objectives of the study. Multiple lines of evidence gathered through a mix of qualitative and quantitative methods allowed for triangulation (i.e. a convergence of results across lines of evidence) and complementarity of findings (i.e. developing better understanding by exploring different facets of a complex issue). The diversity of methods helped compensate for the inherent limitations of each data source and helped mitigate the overall study challenges.

Study challenges included programming complexity, programming changes during the evaluation period, and the limited availability and reliability of data on the funded activities and their performance. While these data were subsequently collected through a client survey, they are subject to the limitations of the survey method and to a reduction in data validity due to the need to recreate data on past performance. Also, since REDOs were not included in the client survey, only a small number of completed planning projects are represented, which limits the assessment of intermediate planning outcomes.

Mitigation measures included the elaboration of program theory to better understand program complexity, the use of case studies representing a range of project success, the contextualization of program changes, a detailed categorization of project types to create a profile of the projects/activities funded, and the development of methods that address gaps with respect to the availability and reliability of performance information.

Strengths of the evaluation design included consideration of the broader program theory, the involvement of key stakeholders during methodology design, and the high client survey response rate (49 per cent overall).

Across methods, close to 300 individuals provided their knowledge and insight to this study, and the findings presented in the report build on a high level of convergence of informed views and opinions. Within this context, the results of the CM and CI sub-program evaluation are deemed reliable and valid. See Appendix B for further detail on the evaluation challenges, mitigation strategies and strengths.

2. Profile of the community mobilization and community investment sub-programs

2.1 Context

Community Development (CD) is one of four programs within ACOA’s Program Alignment Architecture (PAA) (Appendix A) that contribute to achieving the Agency’s strategic outcome of a competitive Atlantic Canadian economy. Through its CD programming, ACOA works with communities and entrepreneurs to stimulate economic growth, improve infrastructure and develop opportunities in local economies, with the desired outcome of creating dynamic and sustainable Atlantic communities. Rural communities are a particular priority of the programming.

CM and CI are two of the four sub- programs that support the goals of the CD program. CM programming supports the mobilization of key stakeholders for planning of economic development initiatives, while CI programming invests in critical economic development infrastructure and other initiatives that support economic development outcomes.

CM and CI programming are delivered using mainly two transfer payment programs: The Innovative Communities Fund (ICF) and the business support element of the Business Development Program (BDP).

The objectives of the ICF are to enhance community or regional capacity through the development of competitive, productive, strategic industry sectors; to strengthen community infrastructure in rural communities to improve their economic development capacity; and to support initiatives that enhance communities’ capacity to overcome economic development challenges and take advantage of their strengths, assets and the opportunities presented. [5]

The objectives of the BDP business support element are to strategically assist not-for-profit organizations to provide specialized services and infrastructure in support of entrepreneurship and small and medium-sized enterprises (SMEs), or to assist local economic development that further advances the Agency’s strategic priorities and the federal government’s regional development policies. [6]

CM and CI programming also support broader ACOA areas of focus such as:

One important recent change to CM programming is the cessation of operational funding to 51 REDOs across Atlantic Canada as of May 2013. Over the evaluation period, REDOs received a substantial amount of CM funding ($29 million) in support of their operational costs, which funded their core regional economic development planning and mobilization activities.

The decision to discontinue REDO operational funding was based on the need to reduce duplication of services, to streamline delivery by ensuring a more direct service to communities and other key clients, and to increase focus on investments that maximize economic growth and job creation. This decision was, at least in part, based on a previous evaluation of the programming that found evidence of overlap between REDO activities and other organizations in some regions. [7] Also, the appropriateness of the REDO model was considered due to the extent of the demographic decline experienced in some rural areas served by REDOs.

2.2 Special initiatives

Over the evaluation period, CM and CI staff also delivered the Atlantic portion of Canada’s Economic Action Plan (EAP) initiatives: the Recreation Infrastructure Canada program (RInC) and the Community Adjustment Fund (CAF). These initiatives aimed to create employment opportunities and address issues of declining infrastructure in communities affected by the global economic downturn. [8]

CM and CI staff also delivered projects in support of Canada’s Roadmap for Linguistic Duality. Specifically, the Economic Development Initiative (EDI) aimed to facilitate sustainable growth in OLMCs, and the New Brunswick Francophone Immigration Project aimed to increase the number of francophone immigrants as well as their social and economic integration into the province. Further detail on the number of projects and amount of funding delivered under special initiatives is contained in section 2.6.

2.3 Program theory

In general, programs are purposeful and organized efforts to intervene in an existing process or situation in order to provide a service or solve a problem [9]. A program’s “theory” is the explanation of how and why a given intervention is supposed to work, taking into consideration the assumptions, causal mechanisms and other factors that contribute to the achievement of expected outcomes [10]. In essence, it explains how a program, within the context of the environment in which it operates, is expected to produce results [11].

2.3.1 Program logic model

A logic model – which sequences the activities, outputs and expected outcomes of a program – is often used to summarize a program’s theory. The logic models for the CM and CI sub- programs are summarized in Table 2.

Table 2: Community mobilization and community investment logic models

Component

Community Mobilization

Community Investment

Activities/Outputs

Consultations Information sessions Maintain/create partnerships Planning Financial support (from ACOA) Coordination

Consultations Networking opportunities Information sharing Investments in support of community projects

Reach

Clients: REDOs; NGOs; non- commercial organizations; local governments; OLMCs; Indigenous communities

Stakeholders: governments; communities; private sector

Clients: geographic communities; sectoral communities (industry sectors); communities of interest (Indigenous, francophone)

Stakeholders: community leaders; financial institutions; private- sector organizations; educational institutions

Immediate Outcomes

Sector development strategies Regional-based strategic plans Increased leadership capacity in communities Improved community collaboration

Increased ability of communities to respond to opportunities and economic crisis Increased economic opportunities Community based partnerships Development of critical economic development infrastructure, including non-physical infrastructure

Intermediate Outcomes

Impacts [V]

Improved community capacity to identify economic development needs and opportunities The implementation of plans and initiatives that have led to economic impacts

Improved capacity to address economic and business development needs and opportunities Economic activities within businesses, communities and industries The creation of economic opportunities that have led to economic outcomes in the community

CD Outcome

Dynamic and sustainable communities for Atlantic Canada

ACOA Strategic Outcome

A competitive Atlantic Canadian economy

Source: Adapted from ACOA’s Performance Measurement Strategy, June 2010.

Broadly speaking, CM planning and mobilization activities were expected to lead to the development of initiatives that result in economic development impacts in the longer term. CI investments in infrastructure and other initiatives were expected to more directly facilitate economic development impacts. Desired impacts were further elaborated through the development of the CED Conceptual/Analytical Framework [VI] (Appendix C), and have been added to the original logic models. The achievement of outcomes and impacts is influenced by factors relating to existing community or industry assets and other contextual elements.

A detailed description of the activities and expected outcomes listed in the logic model is provided below. Further detail on CM and CI inputs and outputs is provided in section 2.5.

Community mobilization

CM activities focus on the mobilization of key stakeholders and on economic development planning. These activities include support to REDOs as well as to industry sectors and communities (including Indigenous and francophone communities) whose activities encompass social, commercial and economic interests.

CM activities are expected to lead to immediate outcomes such as leadership skills in economic development, increased collaboration for economic development, and sector- or region-based economic development plans. Intermediate outcomes include improved community capacity to identify needs and opportunities and the implementation of initiatives that focus on specific opportunities that can lead to economic activities within businesses, industry sectors and communities, and economic outcomes for communities such as investments in infrastructure, increased revenues for businesses and job creation.

Community investment

CI activities focus on investments that facilitate economic development in communities, broader regions and specific industry sectors. CI supports critical economic development infrastructure for communities and industries, marketing initiatives, skills capacity and other types of initiatives. ACOA also works with Indigenous and francophone communities to identify investments that can stimulate transformative change and support strategic development. CI funding is also used to help communities in economic adjustment or transition by providing support in areas that help them respond most effectively to their unique situations.

The initiatives funded through CI are expected to lead to immediate outcomes such as greater capacity of communities to respond to opportunities and economic crisis, partnerships for economic development, and critical infrastructure. Intermediate outcomes include improved capacity to address economic and business development needs and opportunities as well as increased economic activity within businesses, industry sectors and communities, and economic outcomes for communities such as increased revenues for businesses, increased tourism visitation, and creating or maintaining jobs and businesses.

2.3.2 Other influential factors

Literature and key stakeholders consulted in the development of the CED Conceptual/Analytical Framework (Appendix C) pointed to the importance of considering the role of physical, human, social, monetary and natural assets that contribute to and influence the achievement of outcomes along with ACOA support. Other external factors such as global economic forces, population shifts, and environmental changes can also influence the impact of economic development initiatives. These were explored in case studies and were considered in interpreting evaluation findings. The information was also useful for framing current challenges in community economic development, as they relate to programming relevance and need.

2.4 Program accountability and governance

ACOA’s Executive Committee is chaired by the President (Deputy Minister) and is the highest internal governing forum that supports the President in the development of the Agency’s policies and programs. Members of the Executive Committee include ACOA vice presidents (VPs), the Chief Executive Officer (CEO) of Enterprise Cape Breton Corporation (ECBC), and other senior executives. Each regional office is headed by a VP who is responsible for ACOA policy and program activities within their respective region (N.B., P.E.I., N.L., and mainland N.S.). Based in Sydney, N.S., the CEO of ECBC is responsible for the delivery of ACOA programming within Cape Breton, in addition to the Corporation’s own programs.

The Director General (DG) of CD, located at Head Office in Moncton, N.B., together with regional directors of CD, is responsible for CD programming across the Agency. The DG of CD reports directly to the VP of Policy and Programs, while regional directors of CD report to the DG of Operations and the VP of their respective regions. ECBC has a DG of CD who reports internally within the Corporation.

With respect to approval of funding under the ICF transfer payment program, the minister provides direction to the regional VPs regarding approval of projects. Ministerial direction is also sought for some ICF amendments considered to be of a substantive nature, as described in the Agency’s policy on “Amendments to Contribution Agreements.”

In the case of the BDP, project approval authority is delegated to various levels depending on the regional office and type of project, and is also established as per the Agency’s financial signing authority delegation instrument. The BDP levels of authorization are limited by the total amount of the ACOA grant and contribution to a particular project. Within each region, authority for approval levels under $500,000 vary; in general, however, most decisions are made within the delegated authority of the regional VPs. ACOA’s president or VP of policy and programs can approve contributions up to $1,000,000, while the minister approves contributions up to $10,000,000.

Within ACOA, there are functional leads on priorities such as tourism, Indigenous communities, OLMCs and business skills development. These functional leads play a support and coordination role for CM and CI programming that contributes to these priorities.

2.5 Expenditure profile

2.5.1 Proportion of overall agency expenditures

Between 2008-2009 and 2011-2012, CM and CI programming represented 28 per cent of Agency expenditures. Table 3 provides an overview of ACOA’s expenditures by program over the evaluation period.

Table 3: Expenditures ($M) by program, 2008-2009 to 2011-2012

Program

Expenditures

2008-2009

2009-2010

2010-2011

2011-2012

Total

Enterprise Development

215.0

192.2

191.0

172.9

771.1

Community Development

113.1

155.7

161.9

103.8

534.5

Community Mobilization and Community Investment Sub-Programs*  

76.6

126.9

134.7

73.7

411.9

Policy, Advocacy and Coordination

13.3

13.0

13.7

13.7

53.7

Internal Services

0

42.9

41.8

40.0

124.7

Total Expenditures

341.4

403.8

408.4

330.4

1,484.0

Source: Departmental Performance Reports.

*Data for CM and CI came from the GX financial systems.

Note that for 2008-2009, expenditures for internal services in support of program functions were reported within the program areas, rather than separately. Removing the expenditures dedicated to internal services in support of CM and CI programming for 2008-2009 produces an adjusted total of $76.2 million, for an adjusted total of $411.5 million over the four year period.

2.5.2 CM and CI expenditures

As can be seen in Table 4, grant and contribution (G&C) expenditures represent the majority of program spending (92 per cent), followed by salaries (6 per cent) and operating expenditures (2 per cent). G&C expenditures increased in 2009-2010 and 2010-2011 due to the delivery of initiatives associated with the EAP . The slight increase in salaries over this period was also related to the EAP .

Table 4: Annual expenditures ($M) for CM and CI, 2008-2009 to 2011- 2012

Year

G&Cs 

Salaries 

Operations 

Total

2008-2009  

68.5

5.6

1.8

75.9

2009-2010

118.2

6.9

2.0

127.1

2010-2011

127.2

6.2

1.5

134.9

2011-2012

66.4

6.0

1.2

73.6

Total

380.3

24.7

6.5

411.5

Source: ACOA GX Financial Database (Extracted June 25, 2012).

Table 5 details annual G&C expenditures by transfer payment program. Over the evaluation period, BDP expenditures increased by 35 per cent and ICF expenditures decreased by 32 per cent. The majority of CM projects are supported through the BDP; CI projects were supported using funds from both the ICF and the BDP. EDI expenditures in support of the Roadmap for Canada’s Linguistic Duality are included in Table 5 and were delivered using ICF and BDP funds. As mentioned above, CM and CI staff also delivered the bulk of CAF and RInC funding, mostly in 2009-10 and 2010-11.

Table 5: Annual G&C expenditures ($M) for CM and CI, 2008-2009 to 2011- 2012

Year

Expenditures by Transfer Payment Program

Total

BDP

ICF

RInC

CAF

2008-2009 

19.7

48.8

0

0

68.5

2009-2010

27.1

45.6

11.3

34.2

118.2

2010-2011

29.8

31.7

19.4

46.3

127.2

2011-2012

30.2

33.1

3.1

0

66.4

Total

106.8

159.2

33.8

80.5

380.3

Source: ACOA GX Financial System (Extracted June 25, 2012).

2.6 Project profile

To create a meaningful representation of funded projects than was possible through ACOA’s project database (QAccess), the evaluation team undertook an analysis of the 1,087 CM and CI projects supported over the evaluation period, excluding 230 RInC projects ($34.2 million), 167 CAF projects ($82.9 million), 34 EDI projects ($5.19 million) and the NB Francophone Immigration Initiative ($10 million) approved over the evaluation period. The analysis resulted in the creation of two broad categories – planning projects and investment projects. Table 6 shows the number of projects and spending amounts approved for both planning and investment projects over the evaluation period.

Table 6: Planning and investment projects approved, 2008-2009 to 2011- 2012

Type of Community Economic Development Project

Number of Projects

ACOA Approved ($M)

Planning

338

42.6

Investment

749

184.3

Total

1,087

226.9

 

Planning projects included planning for the economic development of a geographic area, industry sector or specific initiatives. Investment projects included funding for: construction or equipment purchases in support of economic development for communities and sectors; marketing initiatives; events; business or technical skills development; and research in support of sector development, among others. The types of projects supported within each broad category will be presented in more detail in Section 4.

3. Findings: relevance

Overall, the evaluation found that the CM and CI sub-programs are relevant activities for ACOA and the federal government. There is a continued need for planning and investment programming as challenges to CED exist to the same or a greater degree than reported in the previous evaluation. In response to changing CED needs, the Agency has made adjustments to programming, including an increased emphasis on investments that generate greater economic impacts and the cessation of operational support to REDOs across Atlantic Canada. CM and CI activities align with ACOA’s strategic outcome and the ACOA Act, and are consistent with the Agency’s roles and responsibilities and broader federal priorities. In general, the Agency’s programming complements rather than duplicates that of other organizations.

In generating these findings, the evaluation team assessed the relevance of CM and CI programming by examining: (1) the continued need for the programming; (2) the programming’s alignment with the Government of Canada’s priorities and with the Agency’s priorities; and (3) federal roles and responsibilities.

3.1 Continued need for programming

Judgment Criteria

Key Finding

The programming-related needs are still present to at least the same degree as they were five years ago.

Most community economic development challenges exist to at least the same degree or are greater than what was reported in the previous evaluation. There was a substantial increase in perceived challenges of an aging population and outmigration since the last evaluation.

Community planning and investments respond to emerging needs.

ACOA’s CM and CI programming responds to emerging needs. Contributing factors include the Agency’s decentralized delivery model; the support provided from the PAC function and ED programming; and stakeholder relationships.

ACOA is aware of changing economic development needs and adjustments to programming are made to meet those needs.

ACOA is aware of changing CED needs and has made some adjustments to programming over the evaluation period. These include the proactive identification or development of projects with greater potential economic impacts and the decision to discontinue operational funding to REDOs.

The roles and responsibilities of other service providers do not duplicate those of ACOA (extent of duplication, overlap or complementarity).

ACOA’s CM and CI programming is unique and complementary to CED programming offered by other organizations. The CM and CI programming’s distinctive traits include its Atlantic-wide scope,; PAC contributions, internal capacity for proposal development and assessment, programming flexibility, and focus on CED goals.

3.1.1 Programming-related needs

Over the past five years, economic development in rural areas has continued to be challenged by the decline of traditional industries, population shifts, infrastructure issues and other factors described below. These issues emphasize the need to support initiatives that increase the competitiveness of Atlantic Canadian rural communities and businesses. Such initiatives include strategic economic development planning and infrastructure required to attract and retain labour; transportation and other types of infrastructure that support economic activity and specific sectors; and marketing and skills development.

Global economy and decline in traditional industries

The global economic downturn negatively impacted traditional industries and employment in Atlantic Canada. By late 2008, Atlantic Canada posted reduced gross domestic product growth. While the Atlantic Canadian economy began to recover in 2010-2011, growth was less stable in 2012 across Canada due, in part, to economic difficulties abroad that affected the global economy. [12][VII]

Increasing global competition from lower cost and/or higher-value production in emerging markets, combined with the economic downturn, negatively affected traditional industry sectors such as fishing, forestry and agriculture, which have long been important in Atlantic Canada, particularly in rural areas. [13], [14] Particular challenges include structural adjustments in the fishing sector, mill closures and job losses in forestry, and the loss of market shares in the agricultural sector. [15] For instance, over a quarter (26 per cent) of clients surveyed for this evaluation stated that industry/company closures are a major challenge to CED, compared to 13 per cent in the previous evaluation. [16]

Company closures related to these challenges contributed to an increase in unemployment following 2008. The Atlantic provinces continue to experience unemployment above pre-recession levels, with the exception of Newfoundland and Labrador, where in 2012, the number of people without a job was lower than in 2008, attributable to growth in the oil and gas and mining sectors. [17] Over half of surveyed clients (52 per cent) reported lack of employment opportunities as important barriers to CED, a significant increase over the previous evaluation (21 per cent). [18]

Demographic challenges, labour shortages and skills

Atlantic Canada’s aging population, out-migration and related lack of skilled workers and volunteers have continued to challenge economic development over the evaluation period. The Atlantic Canadian population is older than that in other Canadian regions due to low birth rates, modest immigration levels and high out-migration rates. [19] Nearly half (48 per cent) of surveyed clients cited an aging population as a key barrier to economic development, compared to 12 per cent in the previous evaluation, while 45 per cent cited out-migration as a key challenge, compared to 15 per cent in the previous evaluation.

These demographic issues have contributed to Atlantic Canada’s growing labour shortages. [20] In a 2011 survey of 600 Atlantic Canadian businesses, 59% reported having difficulty recruiting qualified workers. [21] This issue was more pronounced in rural areas. According to a 2012 report by the Atlantic Provinces Economic Council, Atlantic labour force growth is primarily urban, and the disparity between National and Atlantic participation rates is due to declines in rural regions. [22] The report further states that a specialized skills and expertise are critical to the production of innovative and value-added goods and services, which are necessary for improved productivity and expansion into new markets.

Business skills within SMEs are also critical to the economic growth of Atlantic Canada. Self-employment in Atlantic Canada lags behind other Canadian regions. [23] A 2007 research report by the Institute for Competitiveness and Prosperity suggested that strengthening management skills will lead to greater innovation, productivity, international competitiveness and, ultimately, the prosperity of Canadians. [24] This was echoed by key informants and in case studies completed through this evaluation, where lack of business skills was strongly linked to projects not achieving expected outcomes.

Much like businesses, non-profit organizations need skilled employees and volunteers to manage effectively. According to a 2006 study of non-profit organizations in Atlantic Canada, over half of all organizations in the region reported financial, human and organizational capacity as barriers. [25] The challenge of a skilled labour shortage was also raised by key informants consulted through this evaluation.

Infrastructure issues

Basic infrastructure such as transportation access, water/sewage, broadband and telecommunication, and community/recreational facilities are considered essential to the development of local and regional economies. [26] While the national average age of public infrastructures decreased from 17.5 years in 2000 to 16.3 years in 2007, three Atlantic provinces were among those with the oldest public infrastructures in Canada: Nova Scotia (18 years), Newfoundland (17. 3) and New Brunswick (16.9). [27] Close to half of clients (47 per cent) surveyed indicated infrastructure issues as posing challenges to community economic development, representing an increase from the previous evaluation (13 per cent).

Access to funding

Governments are the single most important source of funding for non-profit and voluntary organizations, accounting for 45 per cent of all funding in the Atlantic region and 49 per cent nationally. [28] As the Canadian economy began recovering from the recession in 2010 -2011, the Government of Canada was reducing spending and provincial governments were also in a period of fiscal restraint. [29] Since municipalities, chambers of commerce, Indigenous groups, social enterprises and other industrial/sector associations often seek provincial and federal funding for economic development initiatives, there were greater challenges in leveraging resources over the evaluation period. Of clients surveyed, 58 per cent identified access to funding as a barrier, compared to 19 per cent in the previous evaluation. Key informants noted that fewer private-sector funds are available to support CED projects in the Atlantic region compared to other Canadian regions due to a reduced presence of private-sector and decision-making centres of larger national or multinational firms.

Francophone and Indigenous communities

As in other regions of Canada, francophone and Indigenous communities in Atlantic Canada experience specific challenges to economic development. According to Statistics Canada data, challenges faced by Canadian OLMCs include greater population decline, out-migration, an aging population, unemployment and low self-employment, and greater employment in traditional industries, which are under pressure. [30] A 2012 evaluation of the ACOA-delivered New Brunswick francophone immigration initiative determined that programming to encourage francophone immigration was necessary to help the Province meet demographic challenges and contribute to its economic, cultural and social future. The recent EDI evaluation concluded that there is an important continuing rationale for programming in support of economic development initiatives for OLMCs. Challenges relating to the economic progress of Indigenous communities include health issues, lower levels of educational attainment, greater unemployment and lower income, reflecting an ongoing need for targeted economic development initiatives. [31]

3.1.2 Awareness of and responsiveness to changing needs

ACOA remains aware of changing economic development needs and opportunities through its decentralized delivery model (i.e. regional and district offices), close partnerships with a variety of government and non-governmental stakeholders, and the knowledge and resources contributed by PAC and ED . In response to changing needs, the Agency has made some adjustments to programming, including the cessation of REDO operational funding, the implementation of the Government’s EAP as well as funding projects with greater potential economic impacts.

Key informants indicate that the Agency’s flexible and decentralized delivery model allows for a better understanding of regional and local issues through direct contact with clients, communities and other key stakeholders such as provincial and local governments. According to key informants, relationships with and contributions from ACOA PAC and ED programming enhance the Agency’s understanding of, and response to, CED challenges. Examples include teamwork between the units aimed at stimulating economic activity in response to a pulp and paper mill closure in Newfoundland and Labrador or maximizing opportunities for a region’s businesses to participate in a global supply chain in New Brunswick.

The Government’s EAP initiatives, RInC and CAF, were delivered by ACOA in Atlantic Canada in response to global economic decline. The programming was mainly targeted at declining infrastructure and other investments to support key industries. As previously stated, in deciding to discontinue operational funding to REDOs, ACOA considered the changing demographics of rural areas and the need to provide greater focus to its other investments so as to maximize economic growth and job creation in response to the global recession and other challenges to economic development. Examples of such initiatives include integrating businesses affected by the decline of traditional industries into the global value chain; technological innovations needed to train the labour force for growing industries; and the development of infrastructure that enables economic development to take place, such as infrastructure related to transportation, electrical upgrades and broadband access..

3.1.3 Extent of overlap, duplication and complementarity with other programming

ACOA’s CM and CI programming is unique and complementary to programming offered by other governments and organizations. While other organizations provide some support for CED, a number of factors contribute to the uniqueness and importance of ACOA’s CM and CI programming in the region, particularly the amount and type of funding support, its Atlantic-wide scope of intervention, its targeted focus on CED, its decentralized delivery model and the coordination with PAC and ED programming.

Other than ACOA, provincial governments are the most regular co-funders of CM and CI projects. Over the evaluation period, the Provinces co-funded 65 per cent of projects, with their contributions representing 26 per cent of total costs. Provincial involvement was similar for both planning and investment projects. Each Atlantic Province has a department that commonly co-funds community infrastructure projects, and all four Provinces have other departments that sometimes contribute funding to sector-specific initiatives. According to key informants, provincial departments often have smaller budgets and less capacity for project assessment and development. Client organizations and other external funding agents confirmed the importance of ACOA’s role in project assessment and development, which often influences the decisions of provincial governments and others to invest in CED initiatives.

ACOA’s programming distinguishes itself from that of other federal government departments because of its specific focus on CED. ACOA’s decentralized delivery model positions it well for delivery of federal economic development initiatives such as the EAP . The Agency’s regional presence allows for better relationships with regional stakeholders and clients and supports consideration of contextual factors. Key informants noted that ACOA plays a key role in fostering collaboration and coordination, as well as the development of strategies and investment projects among federal partners.

ACOA’s unique focus on Atlantic Canada allows it to respond to specific regional needs, opportunities and contextual factors, and to take advantage of synergies between the Provinces. Furthermore, a unique characteristic of ACOA’s CM and CI programming is the ability to draw from PAC research capacity and ED knowledge relating to commercial activity to inform planning, project development and decision making. For example, in Nova Scotia, CD and ED account managers collaborated on an aquaculture association initiative to promote industry best practices and increase community knowledge of aquaculture. The aim was to dispel myths and demonstrate the diversity of aquaculture operations, leading to greater awareness and capacity to pursue commercial opportunities. In Newfoundland, CD and ED programming jointly supported efforts to integrate regional businesses into the supply chain for a major oil and gas project. CI programming contributed to enhancing awareness of opportunities, facilitated networking with major purchasers, and enhanced business skills related to supplying major projects. ED programming supports were then used to work directly with firms to address identified needs (e.g. ISO certification).

As previously mentioned, over the evaluation period, ACOA discontinued operational funding to REDOs. Following the withdrawal of ACOA funding, the landscape of key players in regional economic development planning and mobilization changed and continues to shift. Key informants cautioned that this period of transition will require ACOA to adjust to changes in key stakeholders, and will require the monitoring of the potential risk of capacity gaps.

Organizations that provide advice and training to support entrepreneurs, such as the Community Business Development Corporations, University Business Development Centres in Nova Scotia, and other provincially funded programming, complement some CI initiatives aimed at building the business skills and knowledge of entrepreneurs.

For more detail on other organizations that support similar programming, see Appendix E.

3.2 Alignment with ACOA and government-wide priorities and strategies

Judgment Criteria

Key Findings

There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome.

ACOA’s CM and CI programming aligns with the Agency’s strategic outcome and the priorities of the Government.

 

As indicated in ACOA’s PAA, communities are critical for prosperity. They have varying capacities, challenges and strengths and their economic development requires a flexible, holistic approach. [32] As described in the PAA, by mobilizing key stakeholders for the planning and implementation of economic development initiatives, CM and CI sub-programs support the broader goal of the CD program, which is to foster “dynamic and sustainable communities for Atlantic Canada,” in support of ACOA’s strategic outcome of “A competitive Atlantic Canadian economy”. For example, the evaluation’s project analysis shows that ACOA has supported planning activities related to the economic development of geographic regions and sectors of interest. ACOA investments included initiatives that support the development of growing and traditional industry sectors and community infrastructure, such as high- quality recreational or cultural infrastructure, that attract and retain skilled labour.

According to key informants and project analysis, CM and CI programming has also supported Agency priorities through projects aimed at economic development in francophone and Indigenous communities, projects such as the Celtic Colours Festival and Joggins Fossil Cliffs that support ACOA’s Strategy for tourism growth, and projects that support the development of business skills in Atlantic communities, such as the Black Business Initiative in Nova Scotia, training on innovation and product development in NB and entrepreneurship and business skills promotion in PEI.

With respect to supporting Government of Canada priorities, as previously discussed, ACOA delivered the EAP initiatives, which were a high priority during the evaluation period. There is also explicit and logical alignment between the planning and implementation of economic development initiatives and one of the high-level expected outcomes identified in the Government of Canada’s whole-of-government framework: “strong economic growth,” which is focused on increasing economic growth and development in all regions and all sectors of the economy. [33] ACOA’s programming supports other Government priorities as evidenced by links to speeches from the throne and federal budgets over the evaluation period. For example, ACOA supported the development of traditional industries[34] through investments in economic development infrastructure and training relating to the mining, marine and agriculture sectors. ACOA also supported infrastructure construction[35] related to recreation, transportation and business (e.g. electrical upgrades). Finally, ACOA supported economic opportunities for Indigenous Canadians[36] and skills development and training,[37] as described above, and francophone identity and linguistic duality[38] through the implementation of the EDI.

3.3 Alignment with federal roles and responsibilities

Judgment Criteria

Key Findings

ACOA’s roles and responsibilities in this area of programming are aligned with the ACOA Act.

CM and CI programming aligns with the ACOA Act.

Similar federal organizations in other jurisdictions administer such programming.

ACOA’s CM and CI programming is similar to what is offered through other RDAs.

 

The ACOA Act gives the Agency the authority to “plan, direct, manage and implement programs and projects intended to contribute directly or indirectly to the economic prosperity of the Atlantic region.”[39] By planning and investing in infrastructure and other initiatives that support economic development in rural Atlantic Canada, CM planning and CI investment activities contribute to ACOA’s mandate to “increase opportunity for economic development in Atlantic Canada and, more particularly, enhance the growth of earned incomes and employment opportunities in that region.”[40]

A comparison of ACOA CM and CI programming with that offered by other RDAs also supports CM and CI alignment with federal government roles and responsibilities. As seen in Table 7, all five RDAs deliver services and programming tailored to regional needs and opportunities, with the aim of helping communities adapt to economic change and strengthen their economies. Several RDAs provide non-repayable grants and contributions to not-for- profit organizations through application-based processes to fund activities for strategic/sectoral planning, skills development, community infrastructure and equipment purchasing. In all cases, RDAs provide programming dedicated to official language minorities and Indigenous communities. Other RDAs also deliver community mobilization and planning activities through the national Community Futures Program as well as the aforementioned EDI and EAP initiatives. Most deliver Infrastructure Canada programming.

Table 7: Programming similar to CM and CI delivered by RDAs across Canada

RDA

Type of Programming

Sectoral/regional planning

Marketing

Training /skills development

Community/ recreational infrastructure

Industry/sector infrastructure

Equipment

Tourism

ACOA

X

X

X

X

X

X

X

CED-Q

X

   

X

X

 

X

WD

X

X

X

 

X

X

 

FedDev

X

 

X

X

     

FedNor

X

X

X

 

X

X

X

CanNor

X

X

X

X

X

X

X

RDA

Horizontal Programming

EDI

Indigenous

Community Futures Program

Infrastructure Canada

Support official language minority

Support Indigenous communities

ACOA

X

X

X

X

CED-Q

X

X

X

X

WD

X

X

X

X

FedDev

X

X

X

X

FedNor

X

 

X

X

CanNor

X

 

X

X

4. Findings: performance – effectiveness

Overall, the evaluation found that the CM and CI programming is incremental or critical to the implementation of projects and the achievement of outcomes. Analysis of project data demonstrated that the construction of community, recreational and/or cultural infrastructure is a large component of CI programming, as is support to tourism. CI objectives with respect to these areas are not clearly articulated in program documentation.

The majority of planning and investment projects surveyed achieved immediate outcomes and many investment projects achieved intermediate outcomes. The survey provided some indication of longer- term outcomes achieved, and case studies highlighted the importance of planning to achieving desired outcomes. The evaluation provides evidence that ACOA investments contributed to strengthening the capacity and asset base of communities for economic development, but cannot speak to related impacts on the economic viability of communities. Some barriers to achieving results exist and could be better mitigated, particularly through improved project viability assessment processes. Limitations with performance measurement information present challenges for strategic and efficient program management.

The evaluation team assessed the effectiveness of the CM and CI sub-programs by examining: (1) incrementality; (2) evidence of achievement of expected results; (3) unexpected outcomes; (4) overall achievement of CED impacts; (5) barriers to achieving results, (6) lessons learned and best practices; and (7) evidence that performance measurement is adequate and effective. As previously stated, the evaluation of effectiveness (outcome achievement) does not include programming delivered under the EAP and the Roadmap for Canada’s Linguistic Duality.

4.1 Incrementality

Judgment Criteria

Incrementality Key Findings

The impact of the absence of the programming is similar to or greater than that observed in the previous evaluation, as reflected by:

  • ACOA clients’ opinion of whether the project/initiative would have proceeded without funding; and
  • ACOA’s influence on the involvement of funding partners.
  • CM and CI programming was incremental to the implementation of projects, to obtaining investments from other funding partners and to achieving results.
  • According to survey respondents, the majority of projects would likely not have proceeded without ACOA’s funding. Of those projects that could have proceeded, almost all would have experienced negative impacts, including smaller scope, time delays, lower quality, and difficulty securing other funding.
  • ACOA provides a large percentage of total funding to planning and investment projects and influences the involvement of other funding partners.

 

To gauge the degree to which results being reported are attributable to ACOA’s investments, clients were asked what would have occurred in the absence of the funding received. As shown in Figure 1, almost two thirds of the clients reported that their projects would likely not have proceeded at all (full incrementality). A third of clients reported that their projects would likely have experienced major negative impacts, including problems securing other funding (31 per cent), reduced scope (29 per cent), lower quality (23 per cent) and delays in initiating or completing the project (15 per cent). Only 3 per cent of clients surveyed reported that their projects would have proceeded with only minor negative impacts in the absence of ACOA support.

Figure 1: Incrementality of CM and CI programming

(D)

Full incrementality is slightly higher for planning (68 per cent) than for investment projects (63 per cent). Of investment projects, full incrementality is highest for projects related to construction (68 per cent) and equipment (78 per cent), and lower for events (28 per cent). The degree of full incrementality found in this study is slightly lower than, though still consistent with, what was found in the previous evaluation, in which 69 per cent of investment projects and 73 per cent of planning projects would likely not have proceeded without ACOA’s investment. [VIII]

Another indicator of incrementality is that ACOA influences other partners to support planning and investment projects. The majority (70 per cent) of clients indicated that other funding was secured as a result of ACOA’s decision to provide support. Similarly, 76 per cent indicated that it would have been unlikely or very unlikely that they could have obtained other funding without ACOA’s support. On average, clients reported that 42 per cent of total project funding was secured from external sources as a result of ACOA’s support. Key informants report that ACOA’s investments have an important influence on leveraging other project funding, particularly from provincial governments but also from municipal governments and other federal departments.

4.2 Achievement of expected results

Overall, the evaluation has found that the majority of both planning and investment projects achieved the immediate results expected of CM and CI programming, such as enhanced collaborations and partnerships for economic development. Planning resulted in capacity building and developed plans while investment projects resulted in the creation of infrastructure for economic development and increased capacity for responding to economic opportunities and crises. While it was not possible to comment on the extent of achievement of the intermediate results of planning projects, there is evidence that investment projects contributed to increased tourism visitation, the development of new products, and increased sales and revenues for businesses.

4.2.1 Planning projects

In general terms, planning projects were expected to mobilize partners, build capacity for planning and develop plans for CED. Planning projects undertaken over the evaluation period achieved these immediate outcomes by increasing CED skills and leadership, influencing decision making and leading to concrete actions for economic development.

The client survey response rate for planning projects was good (77 per cent), however the exclusion of REDOs from the survey and the consideration of completed projects only resulted in a small number of responses on outcome achievement (32). Though survey findings on the outcomes of planning projects are not statistically representative of all ACOA planning projects, they provide some indication of outcomes achieved,[IX] and case studies provided valuable information on the importance of planning for the achievement of these longer-term outcomes.

Types of planning projects

There were four main types of planning projects over the evaluation period: 1) support for REDO operations, including core planning activities; 2) other regional economic development planning; 3) industry sector development planning; and 4) studies or plans related to specific projects or initiatives. Planning project proponents included REDOs, industry associations, Indigenous organizations, municipalities and other non-governmental organizations. Table 8 outlines the number of approved projects and amounts of approved funding by project type.

Over half of ACOA approved funding for planning projects (54 per cent) supported REDOs’ operational activities, including their core planning activities. The second highest amount of funding approved was related to planning for the development of industry sectors (17 per cent) such as tourism, renewable/alternative energy, fisheries and agriculture/agri- foods. The greatest number of planning projects (15 per cent of the funding) consisted of project-specific plans or planning/studies that support decision making related to a more specific initiative. These included feasibility studies, needs assessments and engineering or building plans. Finally, 14 per cent of funding supported regional economic development plans that were not part of REDOs’ core operational planning activities. These generally focused on economic development for Indigenous, francophone and other rural communities.

Table 8: Types of planning projects and funding approved, 2008-2009 to 2011- 2012

Planning Project Type and Description

ACOA Projects Approved

ACOA $ Approved

Number

%

$M

%

REDO operational Operational funding for REDOs

89

26

23.0 [X]

54

Industry-related planning Planning for the development of a specific industry such as tourism, oil and gas, or mining

69

21

7.4

17

Project-specific planning Planning for particular initiatives, includes feasibility studies, strategic plans, construction plans and marketing strategies

116

34

6.2

15

Regional economic development planning Planning for community economic development that is not included in REDO operational funding

64

19

6.0

14

Total

338

100

42.6

100

Source: QAccess and detailed project coding data.

REDOs were involved in all types of planning projects in addition to core planning activities undertaken as part of their operations. They acted as proponents for 25 per cent of the other three types of planning projects. REDOs conducted 22 projects (14 per cent of the funding) related to other regional economic development planning, 13 projects (7 per cent of the funding) for industry-related planning, and 26 projects (27 per cent of the funding) for project-specific planning activities. Key informants also suggested that REDOs were collaborators on many types of projects.

Immediate outcomes of planning projects

Judgment Criteria

Key Findings

The contribution of planning activities to immediate outcomes is similar to or greater than that observed in the previous evaluation as reflected by:

  • at least 50 per cent of projects will result in increased community or organization capacity for strategic planning relating to economic development;
  • at least 50 per cent of projects will result in increased leadership for strategic planning relating to economic development;
  • at least 60 per cent of projects will result in improved collaboration relating to economic development within the community or other organizations (stakeholders);
  • 90 per cent of strategic plans have been finalized
  • plans meet “quality” criteria that is more likely to lead to economic development outcomes; and/or
  • strategic plans identify opportunities that can lead to economic impacts such as employment/business creation, sustainability or growth.

There was evidence of the achievement of immediate expected outcomes of planning projects. Of 32 completed projects surveyed:

  • 88 per cent (28 projects) reported improved ability to conduct economic development planning;
  • 81 per cent (26 projects) reported increased business skills;
  • 91 per cent (29 projects) reported improved leadership for strategic planning; and
  • 94 per cent (30 projects) reported improved collaboration for economic development. The average number of organizations involved in a project was19, though the majority of projects involved three to five other organizations.

According to ACOA’s departmental performance reports (DPRs) over the evaluation period, almost all REDOs (92 per cent) finalized strategic plans.

Due to the small number of planning projects that were part of the survey, the evaluation could not conclude on the quality of plans or the proportion of plans that identify opportunities that can lead to economic impacts.

 

The 32 clients with completed plans (representing 64 per cent of invited survey participants) reported the following immediate outcomes either to some extent or to a great extent:[XI]

Less than 1 per cent of respondents reported that the above outcomes were “not applicable.”

According to survey results, 30 out of 32 completed planning projects (94 per cent) resulted in improved collaboration for economic development. The average number of other organizations involved in projects was 19, though the majority of projects involved three to five other organizations. According to client survey responses, provincial governments (93 per cent) were the most frequent collaborator, followed by municipal governments (80 per cent), industry (68 per cent), other federal governments (61 per cent), non-governmental organizations (59 per cent), academic organizations (58 per cent) and Indigenous governments (54 per cent). All case studies highlighted the importance of collaboration as both a facilitator to project success as well as a key outcome.

Case study – planning: enhanced collaboration and the development of plans

ACOA supported the development of a lobster value and marketing strategy in response to challenges faced by the industry in Atlantic Canada.

Engagement and consensus-building activities resulted in increased collaboration among stakeholders in this complex and fragmented industry. The project led to a better understanding of the industry and provided needed direction.

A long-term value and marketing strategy supporting the future competitiveness of the sector was finalized and activities are being implemented.

As reported in the Agency’s DPRs, it met targets for the expected number of collaborators involved in REDO operational planning projects 92 per cent of the time over the four fiscal years covered by the evaluation. [41] The total number of collaborators ranged from 333 in 2011-2012 to 527 in 2008-2009, an average of 6 to 10 collaborators per REDO.

ACOA’s investments in planning projects resulted in the development of different types of plans. As previously stated, analysis of project data reveals that activities focused on the development of regional economic development plans, business plans, marketing strategies, construction plans, event plans and others.

ACOA met targets set for the presence of regionally based economic development plans, as reported by REDO operational projects. The number of plans ranged from 48 to 60 per year over the four years, meaning that an average of 92 per cent of REDOs developed regional economic development plans each year. The evaluation did not directly assess the quality or implementation of these plans.

Of the 32 survey participants that reported completed planning projects, 29 (91 per cent) reported that a plan was finalized.

Intermediate outcomes of planning projects

Judgment Criteria

Key Findings

Plans contain initiatives that have been implemented and that have increased capacity for economic development, as reflected by:

  • jobs created or maintained;
  • reduced out-migration;
  • the creation or survival of businesses;
  • increased sales and revenues for businesses in the region;
  • the attraction of investment;
  • increased exports;
  • the development of new markets;
  • diversified exports;
  • product development;
  • the development of business skills; and
  • tourism visitation influenced by planning activities.

While the numbers are too small to generalize, survey responses for completed planning projects indicate that the plan helped proponents make decisions regarding new initiatives, and increased their ability to respond to opportunities. Plans also resulted in concrete actions, the creation of infrastructure, the acquisition of new financial resources for initiatives, the development of new projects and the creation/ maintenance of jobs. Increased tourism and increased sales and revenues for business were also reported. Outcomes related to exports and reduced out-migration were not commonly reported and may not be highly applicable to planning projects.

 

Survey respondents that had completed their planning projects (32) reported that as a result of their planning activity:[XII]

Survey findings also provided some indication of the types of longer-term outcomes that may be more commonly achieved:[XIII]

Over half of the 32 organizations with completed projects reported that intermediate outcomes related to exports and reducing out-migration were not applicable to their projects. These survey results combined with information collected through case studies suggest that outcomes related to exports and reducing out-migration may not be directly relevant for planning activities.

4.2.2 Investment projects

Types of investment projects

Table 9 presents the types of investment projects that ACOA funded over the evaluation period, based on detailed analysis of project data.

Investment Project Type and Description

Projects

Approved Funding

Number

%

$M

%

Construction/Infrastructure Construction of community buildings, recreational facilities, cultural sites or industry-related infrastructure

291

39

112.3

61

Marketing Promotion of tourism regions/events/attractions and other industry sectors

117

16

26.0

14

Equipment Purchase or lease of equipment for recreational/cultural events and facilities, or industry sector development, training and research

73

10

14.3

8

Events Festivals, conferences and other events relating to recreation/culture, industry sectors or businesses

166

22

13.2

7

Skills Development Training or other initiatives to increase business or technical knowledge and skills

76

10

10.4

6

Research Centre Operation Research centres related to specific industry sectors such as mining and forestry

5

1

4.6

2

Tourism Operation Operation of historic sites, performing arts facilities, museums or interpretative centres

11

1

1.6

1

Immigration/recruitment Investments to attract immigrants and recruit skilled labour

9

1

1.8

1

Total

748

100

184.2

100

Source: QAccess and project coding analysis.

The largest proportion of investment funding was for the construction/renovation of infrastructure (61 per cent). Marketing initiatives represented 14 per cent of investment funding. A large majority of funding for marketing projects was related to tourism, including a $9 million Atlantic tourism marketing initiative. Other types of investment projects included acquiring equipment (8 per cent), implementing events (7 per cent) and developing business or technical skills (6 per cent). Less frequent types of investment projects supported the operation of research centres (2 per cent), the operation of tourism attractions (1 per cent) and immigration or recruitment of skilled workers (1 per cent). See Appendix F for a more detailed analysis of project sub-types and Appendix G for main project types by ACOA region.

Thirty-three percent of investment funding ($61.1 million), representing over half of the funding approved for construction/infrastructure projects, supported community, recreational and cultural infrastructure:

These projects, their expected outcomes and their role in achieving CI objectives are not well articulated in CI programming documents.

Project analysis for the years 2009-2010 to 2011-2012[XIV] also revealed that a large proportion of investment funding was linked to tourism. Many projects aligned directly with ACOA’s Growth Strategy for Tourism (35 per cent, or $49.7 million), or otherwise had implications for the tourism industry (27 per cent, or $38.3 million). [XV] The largest amounts of funding related to the Growth Strategy for Tourism were within marketing ($22.2 million) and construction ($18.4 million) projects. Most of the funding that supported event projects (67 per cent, or $5.7 million) was also related to the Growth Strategy for Tourism. The extent to which CI programming aims to target the tourism sector and its importance in achieving CI outcomes in relation to other types of projects is not clear in CI documentation.

For outside funding that supported tourism and community infrastructure for recreation and culture, a combined 35 per cent ($64.5 million) supported: infrastructure and equipment purchases for industry-sector related training, research and other activities; infrastructure that enables economic activity such as broadband, sewage/drainage and electrical upgrades; buildings that accommodate commercial space; and, training to enhance business or industry-related skills.

Immediate outcomes of investment projects

Judgment Criteria

Key Findings

There is sufficient evidence to argue the contribution of investment programming to the achievement of immediate outcomes:

  • all projects identify collaborators;
  • the average number of collaborators on a project is 2.7 or greater;
  • 50 per cent of projects have resulted in the increased ability of communities to respond to subsequent opportunities and economic crises; and
  • projects resulted in a response to an economic opportunity or crisis.

According to surveyed clients and case studies:

  • 98 per cent of investment projects indicated collaborators’ involvement. The average number of collaborators is 17, with a median of 5 per project.
  • 90 per cent of investment projects respond to an economic opportunity or crisis; 84 per cent report increased ability to respond to subsequent opportunities or crises; and
  • 79 per cent of projects created economic development infrastructure.

 

All lines of evidence show that investment programming leads to immediate outcomes of enhanced collaboration, the creation of infrastructure, and improved capacities. Survey findings below are based on 154 responses (representing 38 per cent of invited survey participants who had obtained funding for investment projects).

Nearly all (98 per cent) of the respondents reported the involvement of collaborators in their initiatives. [XVI] On average, the completed projects reported 17 collaborators per project, with a range of 0 to 1,127. Given the broad range, the median of five collaborators per project is a more meaningful indication of the extent of collaboration.

Respondents reported provincial governments as the most common collaborators (88 per cent), followed by municipal governments (78 per cent), other federal government departments (63 per cent), non-governmental organizations (56 per cent), industry (55 per cent), academic institutions (49 per cent), and Indigenous organizations (37 per cent).

Case studies echo survey findings in that all initiatives included more than five collaborators across a range of organization types, with provincial governments being the most frequent collaborator. [XVII] Furthermore, case studies illustrate the importance of collaborators in achieving results and in the variety of roles they play, whether as funders, as providers of non-financial resources, including expert advice, or as a benefactor.

Survey results suggest that almost all investment projects were developed in response to an economic crisis or opportunity. Ninety per cent of investment clients surveyed reported direct links to an opportunity or crisis. Case studies provide examples of investment projects aimed at developing new industries or mitigating crises related to the economic downturn and the decline of traditional industries.

Case studies – investments: responding to economic opportunities and crises

Several case studies illustrate how, in response to a decline in traditional industries, projects addressed the need to diversify the economy by developing non-traditional sectors in the Atlantic region, such as cranberry production in N.L., e- health in P.E.I., aerospace and defence in N.B. and call-centre support in N.S.

The Celtic Colours festival case study showed how a large festival can address the issue of declining tourism markets, lengthen the tourism season, and build on opportunities related to the unique regional culture and local geography. In 2012, 55 per cent of festival attendees came from outside Cape Breton.

Most investment clients surveyed (84 per cent) reported an increased ability of communities or organizations to respond to subsequent opportunities or crises. Improved capacity to address subsequent opportunities or crises is facilitated by the skills, leadership and experience gained through working with partners to implement initiatives.

A large majority of respondents (79 per cent) reported the creation of physical or non-physical economic development infrastructure. The result was nearly 100 per cent for construction/infrastructure projects. The majority of case studies were linked to the construction or renovation of a building or the creation of non-physical infrastructure such as marketing a festival that brings tourists from outside the region or supporting an industry association to develop a new sector.

Intermediate outcomes of investment projects

Judgment Criteria

Key Findings

There is sufficient evidence to argue the contribution of investment programming to the achievement of intermediate outcomes: jobs created or maintained; reduced out-migration; creation or survival of businesses; increased sales and revenues for businesses in the region; attraction of investment; increased exports; new markets developed; diversified exports; product development; business skills development; and tourism visitation.

The most frequent impacts of investment projects are increased tourism visitation, the development of new products, and increased sales and revenues for businesses. Over half of proponents surveyed said projects resulted in the creation or maintenance of jobs, led to new financial resources for initiatives and new investments, and/or created or maintained businesses. Investment projects did not often lead to impacts related to exports or reduced out- migration.

 

Investment projects often contribute to the achievement of intermediate outcomes, most notably increased tourism, the development of new products, and increased sales and revenues for businesses. According to survey results, tourism impacts were the most common intermediate outcomes, with a significant majority (79 per cent) of respondents reporting increased visitation – a finding that reinforces the extent of support for tourism suggested by the project analysis. Case studies illustrate how investments in community recreational infrastructure can support tourism, and how support for a major tourism festival can increase external visitation.

Case studies – investments: increased sales and revenues for businesses

In 2011, the Celtic Colours International Festival in Cape Breton resulted in $472,837 in direct ticket sales and an estimated $8.1 million in audience expenditures in the region on accommodations, food, art, transportation and retail.

Upgrades to the Yarmouth Mariners Centre contributed to increased facility rental revenues, from $50,000 in 2009 to $125,000 in 2012. The project also contributed to Yarmouth being able to host the World Hockey Junior A Challenge in 2012 and 2013; this event generated $2.6 million of economic activity in the Yarmouth area in 2012 alone.

A large majority (78 per cent) of respondents reported increased sales and revenues for businesses. Again, case studies illustrate how the development of an industry, the creation/expansion of a tourism event, or community infrastructure can increase revenues to businesses.

A significant majority (78 per cent) of respondents also reported the development of a new product. These were often new tourism products such as tours, festivals and other events.

According to survey results, jobs were created or maintained by 70 per cent of investment projects. Of the 78 respondents that reported job figures, 33 per cent created one to two jobs and 30 per cent created three to five jobs. With respect to supporting continued employment, 36 per cent of respondents reported that their projects contributed to maintaining one or two jobs and 26 per cent of projects reported contributing to maintaining three to five jobs.

Case study – investment: creating or maintaining jobs and businesses

The Membertou Economic Benefits Office was supported to help facilitate the participation of Indigenous businesses and individuals in the Sydney tar pond cleanup exercise.

Approximately $49 million in contracts for the clean-up exercise were awarded to six businesses, which created 69 jobs. The tar pond clean-up was successfully completed and the organization has moved on to support Indigenous business and job growth in relation to other projects.

Survey findings also suggest that close to two-thirds (60 per cent) of investment projects resulted in the creation or maintenance of businesses in communities. Of the 49 respondents reporting business impacts, about 50 per cent stated creating one to two businesses and a further 10 per cent reported three to five businesses; 25 per cent indicated that no businesses were created. Almost half of the 55 respondents reporting that their project had contributed to the maintenance of businesses indicated that one to two businesses were maintained; 27 per cent indicated that three to five businesses were maintained.

Over two thirds (67 per cent) of respondents reported obtaining new financial resources for initiatives as a result of their project, which may have been from other levels of government or private sources. Over half (61 per cent) of respondents reported that their project led to obtaining new investments in communities or regions. Case studies demonstrated how multi- purpose community centres can result in the building of new hotels or other infrastructure, adding to the economy of the communities. In the case of a multi-purpose centre in Grand Falls, a new hotel and gas station were developed as well as a new residential sub-division, expanding the tax bases and amenities for the region.

Case study – investment: developing business skills

In response to industry research suggesting a need to further develop tourism products/experiences in Atlantic Canada to meet market demand, ACOA supported a project by the Gros Morne Institute for Sustainable Tourism to deliver workshops on experiential tourism, best practices in the use of online tools, and sustainable tourism practices to 1,249 tourism operators across Atlantic Canada.

Facilitators with tourism expertise in each region were trained to deliver the workshops and to tailor them using local content, which built capacity for future support to the industry.

In a follow-up survey, 67 per cent of participants reported using the knowledge to make changes in marketing, use of technology or product development.

Almost half of the investment clients surveyed (48 per cent) reported the development of new markets. Case studies of the cranberry industry in Newfoundland and Labrador and the aerospace and defence industry in New Brunswick provide examples of how projects can impact the development of new markets. The case study of the Celtic Colours International Festival also demonstrated how it developed new markets for local Celtic musicians through links to other international festivals.

According to a detailed project analysis, 49 of the 749 projects were dedicated to building business skills in the areas of management, productivity, market readiness, innovation and commercialization. The case study of the Gros Morne Institute for Sustainable Tourism provides an example of how ACOA contributes to building business skills within a given sector.

Outcomes related to exports and to reducing out-migration were not as commonly achieved as others. Only 28 per cent of investment respondents indicated success in diversifying exports (55 per cent stated it was not applicable), and only 23 per cent indicated success in increasing exports (51 per cent stated it was not applicable). Only 27 per cent of respondents reported reducing out- migration (40 per cent stated it was not applicable). These findings suggest that outcomes relating to exports and out-migration might not be directly relevant for investment projects.

4.3 Unintended outcomes of planning and investment programming

Judgment Criteria

Key Findings

Not Applicable

There were three main unintended outcomes of programming that should be considered in program descriptions as well as project viability assessment processes:

  • enhanced community capacity and quality of life;
  • spinoffs; and
  • negative consequences for organizations and communities.

 

The evaluation identified three categories of unintended outcomes of CM and CI programming: increased community capacity and quality of life; economic spinoffs; and, negative consequences. These outcomes are not included in the existing program logic models but are identified in the CED Conceptual/Analytical Model. They are important outcomes to consider in clarifying the types of projects funded and the expected results of programming as well as in mitigating risks.

Case studies – investments: quality of life and spin-offs

Projects contributing to multi-purpose centres in Yarmouth, N.S., Grand Falls, N.B., and Shediac, N.B., showed that infrastructure supports local businesses as well as cultural and sporting activities in the community and is seen to contribute to a greater standard of living.

These investments can be strategic in retaining labour force provided that there is sufficient employment in the area to meet income needs and allow for resident use. They also result in spinoff revenues generated for restaurants, catering companies, accommodations, etc.

Informants emphasized that the improved amenities and services increased pride among residents and hope for the future of their communities and their ability to overcome economic challenges.

Community capacity and quality of life

Case studies and key informant interviews frequently highlighted outcomes related to improved community capacity and quality of life from having better amenities. Improved community resources, including new multi-use facilities or recreational centres, can be used to retain and attract residents as well as contribute to feelings of community connectivity, pride and hope.

Spin-offs

According to case studies and key informant interviews, there were several examples of economic spinoffs as a result of planning and investment programming:

Negative outcomes

Case studies and interviews suggested that weak project planning or organizational capacity can result in the inability to meet expected outcomes, and in financial stress for the proponent and partners. For example, case studies indicated that events that did not meet expected ticket sales or community buildings that did not meet rental targets caused organizations to take on additional debt or to close entirely.

There was also evidence of stress on volunteers with greater than manageable expectations in terms of time and duties. Availability of volunteers and volunteer burnout is becoming a greater issue with an aging and declining rural population.

Case study – investments: weak planning and management capacity affecting project outcomes

A case study that examined the construction of a multi-purpose centre shows that outcomes were negatively affected by weak board capacity, poor planning and volunteer burnout. The structure did not meet the needs of the target market. Since it did not generate the revenues expected, its continued maintenance resulted in further stress to the non-profit organization and its volunteers.

4.4 Roadmap for linguistic duality initiatives: achievement of outcomes

As previously mentioned, between 2008-2009 and 2011-2012, ACOA CM and CI staff approved 34 projects for $5.19 million in approved funding in support of the EDI. A 2012 Summative Evaluation of projects delivered as part of the EDI reported the achievement of the following immediate outcomes: development of new expertise through innovation; diversification of activities; partnerships; and increased support for small businesses. The evaluation also concluded that the achievement of intermediate results was strong and demonstrated significant capacities built, and enterprises and communities developed. Unintended outcomes included the development of closer relationships with OLMCs and higher than expected leveraging from other organizations. [42]

The 2012 evaluation of the $10 million New Brunswick francophone immigration initiative funded under the Roadmap reported that the initiative conducted promotional and awareness activities with employers, communities and the general public. The report stated the initiative resulted in greater than expected immigration numbers to date, and is well-positioned to achieve the expected outcome of increasing immigration though the time frame did not allow for a full assessment of outcome achievement. [43]

4.5 Overall achievement of CED impacts

Overall, the evaluation provides evidence that the capacity and asset base of communities is strengthened as a result of planning and investment funding offered by ACOA and its partners. Planning and investment programming provides access to capital that contributes to planning for economic development and creating important economic infrastructure and other supports. However, the evaluation cannot speak to the overall impact of this strengthened asset base on the economic viability and sustainability of communities.

The evaluation lends support to the activities, impacts and contextual factors outlined in the CED Conceptual/Analytical Framework (Appendix C). The framework illustrates the importance of the capacity and asset base from which communities draw in order to: respond to economic opportunities and threats; increase economic stability and sustainability; and, improve the quality of life of residents. Furthermore, the evaluation confirms many of the contextual challenges or barriers to the achievement of outcomes, detailed in the next section as well as in the section on program relevance.

4.6 Barriers to the achievement of planning and investment outcomes

Judgment Criteria

Key Findings

Evidence of factors that are impeding the success of the CM and CI sub-programs are known, and mitigation strategies are implemented where applicable.

Three key factors impeded the achievement of planning and investment outcomes:

  • capacity and business skills:
    • project proponent: governance, planning and risk management, leadership, business skills, partnerships, project management, volunteers
    • ACOA: insufficient proposal assessment, gaps in knowledge sharing across teams
  • contextual issues, most notably the economic downturn, community and industry assets and distance from markets for many regions
  • funding issues, including the availability of funding, project costs and approval processes.

While some mitigation strategies for management capacity and funding barriers were identified, they were not consistent across the Agency.

 

Case studies and client survey data were the primary source of information relating to barriers and facilitators of project success. Key informant interviews, particularly with ACOA staff and management, also identified internal challenges. Both barriers and facilitators fell into three categories: capacity and business skills, contextual factors, and funding. Mitigation strategies were not consistently implemented across the Agency.

Capacity and business skills

Case studies and key informants identified issues related to capacity and business skills on the part of the project proponent or the Agency. For the project proponents, weak governance structures, planning and risk management practices, leadership, business knowledge and skills, including project management, lack of strategic partnerships and limited volunteer capacity led to a greater risk of not achieving project outcomes. ACOA interviewees indicated that governance training was provided as needed to organizations experiencing difficulties with their board or overall management. However, training was generally on a case-by-case basis and was not carried out by all regions.

ACOA’s capacity to assess projects also affected project outcomes. Several case studies provided evidence that inadequate project viability assessment and premature project approval can contribute to less successful projects and weaker outcomes. Key informants highlighted the importance of sharing information and looking for synergies with the other ACOA units, PAC and ED . Early concept vetting committees, which include representatives from other units, and related processes have been implemented in some regions and show promise in mitigating some internal management challenges.

Contextual factors

The evaluation identified numerous contextual factors through survey data, key informant interviews and case studies that can impact the achievement of project outcomes:

Case study – investments: increasing competitive advantage

In the case of the renovation of a large historic building to develop the e-Health industry in Summerside, P.E.I., the lack of local amenities, including a local airport with regular flights, and the absence of training in the community were reported barriers to project success.

ACOA worked with the province and other partners to attract new companies to the building and provide refit costs. The Agency also collaborated to bring a specialized training program through Holland College to Summerside, mitigating issues related to lack of skilled workers.

The Agency can play a role in levelling the playing field for business attraction and industry sector growth. Some case studies show that ACOA has considered the lack of particular community and industry assets in its project development and review processes to a certain extent (see example in above box).

Funding

As stated in section 3.0, funding was the most important barrier to project success according to surveyed clients. The availability of funding is often tied to the economy and was frequently linked to the capacity of provincial governments and other funders to contribute to projects.

Delays in funding approval processes can also influence project success. Key informants spoke of cost overruns that can occur, for example, if construction is delayed to winter months or if construction costs increase over the delay period.

4.7 Lessons learned and best practices in delivery

Judgment Criteria

Key Findings

Not Applicable

Key lessons learned and best practices:

  • The strong assessment of proponent organizational capacity, funding and business plans by ACOA decreases risks and improves the achievement of outcomes.
  • Short-term programming such as the EAP and CIIF can cause confusion among clients and other stakeholders about regular programming parameters. Short-term funding can also deplete the funds available from partners and potential project proponents, affecting the number and quality of project proposals through the regular planning and investment programming.
  • Approval delays are detrimental to projects in terms of missing opportunities and facing cost overruns.
  • Self-sustainability may not be a reasonable expectation for all projects, particularly those that support groups or communities that experience greater financial or other challenges and are unable to cover costs.
  • Regional approaches to planning and investment projects align with the notion of functional communities and offer opportunities to develop community partnerships and make the best use of limited funds.

 

Lessons learned

Key informant interviews and case studies reveal important lessons learned and best practices for planning and investment programming that can be considered for program improvement. Perhaps the most important lesson learned is the need for a strong, consistent assessment of project viability and risk from the outset in order to reduce risks and improve outcomes. There are promising regional tools and processes for project viability assessment, such as the use of project concept vetting committees.

Initiatives aimed at building economic development capacity for minority groups such as Indigenous and less lucrative or emerging sectors may have fewer assets to contribute and require more support to implement economic development projects, particularly if they cannot obtain much support from other funding organizations. If these sectors or groups are determined to be Agency or regional priorities for development, their existing assets and the availability of other funding or lack thereof should be considered in determining the extent of support that ACOA will provide, and whether operational/administrative activities should be supported.

With respect to EAP programming, key informants identified two lessons learned. First, while it offered many opportunities to communities, EAP projects also often depleted potential project proponent and partner funds, affecting their ability to engage in other economic development projects. Second, because EAP programming was managed by the Agency and had different funding requirements and parameters from regular programming, it caused some confusion among clients and partners in terms of what could be funded through core programs.

Best practices

According to program key informants, collaboration of RDAs in the development of new programs was a best practice that resulted in efficiencies through sharing of responsibilities attached to the development of performance measurement strategies, TB submissions and sharing of best practices. Another best practice was the use of ACOA’s existing terms and conditions for administering CAF. According to key informants and a literature review, an important best practice is striving for regional approaches to CED in order to maximize the impacts and reach of investments, especially in the context of demographic changes and population movements. Several case studies demonstrated the benefits of communities collaborating on infrastructure such as tourism events and multi-purpose centres. These initiatives benefited from shared resources, including funding, infrastructure and effort, while serving a greater number of people in many cases. Supporting Atlantic-wide delivery of a high-quality business training course can also be seen as a best practice when it allows for cost savings in course development and it meets a common need.

4.8 Adequacy of performance measurement

Judgment Criteria

Key Findings

Evidence that CM and CI PSA performance measurement (PM) is adequate and effective in reporting on the achievement of outcomes. PM information is used by ACOA and ACOA-funded organizations to support decision making. Performance information is available for use in the evaluation.

PM was not adequate or effective in reporting on the achievement of outcomes. PM information was used for DPR reporting, including the: number of projects, amount of investments, number of plans completed and number of partners involved in planning projects. The Results Management Tracking System (RMTS) is not used consistently, though it is promising as a tool to track outcomes. PM is used to some extent by ACOA management to support decision making, but aggregate data was limited. The evaluation unit used DPR data and financial information as well as project final reports when available. Project coding through QAccess was limited in providing information on the types and scope of projects funded.

 

PM data is used to some extent by ACOA to aid program management. ACOA uses high-level dashboard data to monitor program activity such as the number of projects and funding amounts. Management uses QAccess queries and project reports (when available) to establish project precedent in terms of activities and/or outcomes and to inform decision making. However, key informants suggested this does not adequately support strategic program management. PM data collected for DPRs is limited to the number of plans and number of partners for CM programming, and to the number of projects and amounts funded for CI programming. There is limited reliable, aggregate-level data on the nature of approved projects and their outputs, and the achievement of project outcomes.

More consistent use of available QAccess fields would enable the capturing of useful information for strategic program management, such as the number of strategic/proactive investments and projects that support cross-Agency priorities such as tourism, marketing and skills development. However, existing QAccess fields are limited in capturing some important project types such as infrastructure aimed at community amenities with recreational or cultural ties; infrastructure aimed at supporting certain industry sectors; and investments in events that support economic development.

Though some financial data were available and used in the evaluation, particularly from ACOA’s GX and QAccess systems, substantial resources had to be invested in the production of reliable project information due to issues with project coding, inconsistent project summary form content, the inconsistent collection and uploading of final reports and other limitations. ACOA’s RMTS, which will allow account managers to track outcomes, holds promise for the availability of outcome data but is not yet fully implemented.

The majority of clients surveyed reported that they were collecting and monitoring performance data (94 per cent) and using the information for decision making (87 per cent). Several case studies demonstrated PM data was being used for reporting economic impacts and other outputs and outcomes to funders.

5. Findings: performance – efficiency and economy

Overall, the CM and CI programming was delivered in a cost-effective manner. Internal costs were proportionate to the amount of funding delivered and compared reasonably between ACOA regions. ACOA support enabled projects to leverage substantial funds from other organizations. Mechanisms that fostered efficient and economical delivery included: enhanced planning processes; efforts to achieve greater economic impacts; collaboration among ACOA units; and, coordination with other RDAs in the delivery of national initiatives.

The evaluation assessed efficiency and economy by examining (1) the utilisation of resources, (2) evidence of mechanisms that support efficiency and economy, and (3) consideration of alternative delivery mechanisms and best practices that resulted in efficiency and economy gains for ACOA.

5.1 Efficient utilisation of resources

Judgment Criteria

Key Findings

Delivery costs across regions reasonably compare to one another within the context of factors affecting regional delivery.

Delivery costs are proportionate to the amount of funding and number of projects delivered. ACOA was efficient in delivering EAP programming. Amount of G&C delivered by salary dollar and by operational dollar compared reasonably between regions.

ACOA has in place mechanisms to ensure that the most efficient and economical means are being used to administer the programming.

Internal tools and mechanisms allow for strategic decision-making in the selection of projects that result in critical economic infrastructure (or non-physical infrastructure) being developed.

Mechanisms that support efficient and economical delivery include regional strategic planning exercises to help focus investments, completion of feasibility studies prior to investments and leveraging of external funds, which has increased since the previous evaluation. ACOA does not have a reliable process in place to regularly track types of investments funded, which challenges its ability to manage the program strategically and efficiently. The extent of coordination between units or programming areas that supported similar projects was unclear. Most regions have increasingly focused on developing and approving projects with the greatest potential for achieving economic impacts. Tools and mechanism that support these efforts include the development of regional community investment plans and concept vetting committees. The use of such tools and mechanisms is not consistent across regions.

5.1.1. Delivery costs

Delivery costs were generally proportional to the amount of programming delivered and compared reasonably across regions over the evaluation period.

In order to provide the most accurate picture of expenditures specific to CM and CI sub-programs, the cost of internal services, Treasury Board-funded costs and other statutory and miscellaneous expenditures have been excluded from subsequent analyses. These costs amount to $39.5 million over the four-year period from 2008-2009 to 2011-2012, or an average of $18 million a year.

Expenditures for CM and CI programming over the evaluation period amounted to $411.5, with $380.3M (92 per cent) in G&Cs, $6.5M (2 per cent) in O&M costs and $24.7M (6 per cent) in salaries. Figure 2 represents program expenditures by region, with ACOA NB showing the largest amount of total expenditures, followed by NL, NS, PEI and CB.

Figure 2: Community mobilization and community investment expenditures by region (2008-2009 to 2011-2012)

Source: GX Financial Data, August 2012; ECBC financial records, December 2013.

(D)

The amount of G&Cs delivered varied by region due to factors such as regional demand, geographic coverage, population served, and regional priorities. When considering all G&Cs expenditures including the EAP , ACOA NB delivered the most programming ($108.8M), followed by NL ($99.2M), then NS ($76.6M), PEI ($53.3M), CB ($39.5M) and ACOA Head Office ($2.8M).

Salary expenditures associated with delivering the programming compared favourably with the previous evaluation, representing a slight decrease from the previous 6 year period (5.8 per cent vs 6.2 per cent) due in part to the additional delivery of EAP initiatives, the Recreation Infrastructure Canada (RInC) and CAF. These costs supported activities such as working with clients on proposal development, proposal assessment and approval processes, processing financial claims, monitoring, as well as other activities.

Annual salary costs per region were reviewed (see Appendix H for more detail). A general increase in salary expenditures in 2009-2010 is attributed to account manager and support staff time dedicated to delivering EAP initiatives. The ratio of G&C funding delivered per salary dollar helps interpret regional differences. These ratios compared reasonably among ACOA regions with CB delivering $18.29 in G&C per salary dollar, followed by $17.90 for NB, $15.70 for NS, and $14.10 for NL. PEI delivered $26.60 ; this region covers less geography and operates with a smaller staff contingent, making for a reasonable comparison. The calculation of G&C per salary dollar was not considered for HO recognizing that the majority of staff activities do not focus on the direct delivery of G&Cs but are dedicated to program governance, coordination and reporting along with other corporate functions that support the programming such as internal human resource and finance services.

Beyond salary expenditures, other operational costs were reasonably consistent across regions. Figure 3 presents operational spending by ACOA region, including costs associated with communications, travel, and professional services such as conducting environmental assessments required for infrastructure projects. Ratios of G&C dollars delivered per operational dollar help to contextualize the regional differences. For the most part, ACOA regions compare reasonably well with CB delivering $108.36 in G&Cs per operational dollar spent, followed by NB ($85.10), PEI ($69.55), and NS ($54.40). NL is lowest ($42.64), in part due to the increased cost of travel and professional services associated with its greater geographic coverage. The specific expenditures by type of operational activity for CB region were not available but these included activities such as travel for client monitoring and training, rentals, and other activities that supported the delivery of the programming.

Figure 3: Operational spending per region

Source: GX Financial Data, August 2012;ECBC financial records, December 2013.
 
(D)

5.1.2 Mechanisms that support efficient and economical delivery

Multiple mechanisms currently exist to support efficient and economical program delivery. These planning, governance and delivery mechanisms were often aimed at selecting and supporting projects with greater potential of achieving economic impacts. The use of these tools and mechanisms varied between regions.

Planning mechanisms:

Governance mechanisms:

Delivery mechanisms:

5.2 Alternative modes of delivery for increasing efficiency and economy

Judgment Criteria

Key Findings

Program management considers alternative modes of delivery outside the organization

Alternative ways of increasing efficiency and economy were considered and some were implemented, including the use of pan-regional management committees and efforts to fund projects with greater economic impacts. However, key informants caution that proactive development requires more resources and can carry greater risks related to ACOA’s increased influence on projects, and increased perceived accountability for project results

The decision to discontinue operational funding of REDOs reduces overlap in services with other organizations and is expected to allow ACOA to provide more direct client services.

A number of suggestions were made to improve efficiency including increased collaboration between ED , CD and PAC, and increasing client awareness of programming goals to improve proposal quality.

The organization considers its own best practices in implementing alternative approaches for increased efficiency. Best practices were identified and implemented.

 

Alternative ways of achieving outcomes were considered and some actions were taken to increase efficiency and economy. The most significant change in delivery is ACOA’s decision to discontinue operational funding to REDOs for regional planning and mobilization is expected to result in further efficiencies but again, the shifting landscape in key players carries a risk of reduced capacity to coordinate and implement initiatives. Also as stated in the previous section, pan- regional management committees allowed for exchanging best practices and coordinating delivery of CI projects. Several other practices were considered and to some extent implemented to help maximize economic impacts of projects such as harnessing resources of ED programming and PAC units, strategic planning efforts and concept vetting processes. Key informants cautioned that efforts to develop projects with greater potential for economic impacts require more internal resources and time during project development stages. In particular, they spoke of the time needed to for both internal and external coordination with key players and capacity building with clients. Also, proactive project development may hold greater risks in terms of ACOA’s perceived accountability for increased influence over the nature of projects, particularly considering ACOA’s incremental role in obtaining funds from other organizations.

Key informants suggested possible improvements to increase efficiency, including: promoting program goals to potential client groups to maximize proposal quality; further collaboration between units for project development; and better assessment of proposal/concept viability and client capacity to implement and sustain initiatives where applicable. Instances were cited where lack of such scrutiny allowed for investments in some weaker projects, negatively affecting efficiency and economy. It was noted guidelines were being developed to improve project assessment, but were not fully implemented. Other suggestions included the need to streamline ICF approval processes to the extent possible, to reduce approval delays. As previously mentioned, delays in project assessment and approval have resulted in missed opportunities for promising initiatives and/or increased cost (e.g. winter construction). Also, the availability and use of better information on program activities and outputs could enhance strategic decision making, as stated in section 3.

6.0 Conclusions and recommendations

Based on the findings presented, the evaluation identified 7 conclusions:

  1. ACOA’s CM and CI programming responds to a continued need and is aligned with GoC priorities (i.e. EAP , Roadmap for Linguistic Duality), and the Agency’s priorities and mandate (i.e. economic competitiveness and growth; ACOA’s Growth Strategy for Tourism). ACOA is aware of changing CED needs; some adjustments to CM and CI programming were made to address changing needs including the cessation of the REDO model and efforts to support projects with greater economic impact potential.

  2. ACOA CM and CI programming complements rather than duplicates that of other partners. However, a collaborative, coordinated and strategic approach among partners is critical to the achievement of CED outcomes. ACOA works alongside a variety of stakeholders (who often co-fund projects) towards these outcomes. The dissolution of the REDO model brings about a necessary period of transition as ACOA and CED partners realign resources and clarify roles and responsibilities. During the period of transition, there is a risk of capacity gaps for regional-level planning or other economic development activities.

  3. ACOA planning and investment program funding was incremental to the implementation of projects, to obtaining investments from other funding partners, and to achieving outcomes. Planning projects generally achieved immediate outcomes by increasing CED skills and leadership, influencing decision-making and leading to concrete actions for economic development. It was not possible to conclude on the overall achievement of intermediate outcomes of planning due to low survey representation of projects with completed plans and strategies. However, case studies supported the importance of thorough project planning for greater economic development impacts. Investment projects generally achieved immediate outcomes such as creation of infrastructure for economic development and increased capacity for responding to future opportunities. Intermediate outcomes included increased tourism visitation and sales and revenues for businesses, as well as the development of new products. There was evidence of modest job creation and maintenance as well as modest business creation and survival.

  4. A third of investment funding supported the development of infrastructure for community, recreational and cultural activities. These investments can be useful in providing amenities that attract and retain residents/labour force, where sufficient employment opportunities exist. The role and relative importance of these activities for achieving CI outcomes is currently not well articulated in program documentation. Similarly, 35 per cent of funding directly supported tourism, but the extent to which CI programming aims to support tourism relative to other sectors or other types of activities is not clear. Outside community/cultural/recreational and tourism-related projects, a combined 35 per cent of investment funding supported: infrastructure and equipment purchases for industry-sector related training, research and other activities; infrastructure that enables economic activity such as broadband, sewage/drainage and electrical upgrades; buildings that accommodate commercial space; and, training to enhance business or industry-related skills.

  5. An analysis of facilitators and barriers to the achievement of outcomes as well as lessons learned demonstrates the importance of considering management/delivery factors (i.e. internal – project viability assessment; incorporation of knowledge from ACOA PAC function; external – client skills and capacities), contextual factors (i.e. global economic decline, deteriorating infrastructures), and funding (i.e. availability and timely approval processes). While there was some evidence of the use of strategies to mitigate barriers, it was not clear that knowledge related to common barriers to achieving outcomes is used in the selection or monitoring of projects.

    ACOA management uses some performance measurement data pertaining to expenditures and number of projects by transfer payment program to support decision-making. Some PM data was used in the evaluation (e.g. DPR , Q Access, GX), however a detailed analysis of all projects was required to produce information on activities funded. There continues to be challenges with the adequacy of information available on activities, outputs and outcomes for individual projects and at the aggregate level; these challenges present barriers to strategic and efficient program management.

  6. ACOA’s CM and CI programming was delivered in a cost-effective manner. Internal costs are proportionate to the corresponding level of funding involved and number of projects delivered. Leverage of other funding has increased since the previous evaluation. ACOA was efficient in delivering substantial EAP programming without additional dedicated resources; however, this resulted in fewer projects related to ACOA’s core CM and CI programming being delivered.

  7. Several regions have some promising practices that aid in the selection of investment projects that achieve greater economic impacts such as well-informed investment strategies and concept vetting committees. Proactively developing projects with greater impact potential holds promise in increasing selection and funding of projects that maximize economic development impacts but may require more internal resources. Risks associated with increased influence over funded initiatives could emerge.

    Some gaps exist in the consistent application of strong proposal viability/risk assessment processes to reduce risks. The extent of coordination between tourism, business skills development, trade and innovation projects with other similar ACOA programs was unclear, which may create inefficiencies in achieving Agency objectives relating to those areas. Currently, there is a lack of consistency in tracking performance reporting and evaluation of these projects across the Agency.

Overall, the evaluation’s recommendations are aimed at improving programming by balancing a consistent, targeted approach with required flexibility; improving the availability and access of information and knowledge to manage for results; and building on existing and new relationships, expertise and intelligence to narrow the gap between ACOA’s community economic theory and practice, as implemented through the CM and CI sub-programs.

Recommendation 1:

In order to ensure a clear corporate direction for programming while allowing for regional variability and flexibility, ACOA should build on its corporate knowledge, lessons learned and current best practices in CED to formulate a Community Investment framework.

The framework should outline the Agency’s approach to planning and investment programming (including main project types and expected outcomes), while acknowledging regional context, plans, priorities and best practices. It should also describe the Agency’s approach to engage, coordinate and collaborate both internally and externally (partners, subject matter experts) in the achievement of programming outcomes.

The following is a list of considerations in developing the framework that build on evaluation findings:

Recommendation 2:

In support of results-based management, decision-making, and the strategic coordination of initiatives that cut across programming areas (e.g., business skills development, tourism, trade, innovation), ACOA should ensure that quality and timely information on activities (i.e. main project types), and outcomes are systematically collected on file and in electronic systems.

While new tools may be needed, consideration should be given to consistent implementation and use of existing processes, tools and systems (i.e. PSFs, Q Access coding, results tracking, collection of final reports, 2-year follow-up).

It is expected that the systematic collection and reporting on activities and outcomes of CI initiatives that cut across programming areas will help maximize synergies and minimize the risk of under-representing Agency efforts during evaluations and other performance reporting exercises.

Recommendation 3:

In an effort to increase efficiency and effectiveness of programming and mitigate risks of not achieving expected outcomes, ACOA should build on current efforts to develop and systematically implement a standardized process/tool to assess project viability.

The process/tool should assess the likelihood of achieving outcomes (e.g. market response, use of service or infrastructure), as well as the human (project management/governance, involvement of the right partners), financial, and infrastructure capacity of the proponent organization or community to conduct the work outlined in the proposal and to sustain the achievement of outcomes and impacts following project completion.

Figure 4: Alignment between CM and CI evaluation conclusions and recommendations

(D)

Appendix A: Program alignment architecture chart

Strategic Outcome

Program

Sub-Program

A competitive Atlantic Canadian economy

Enterprise Development

Innovation

Entrepreneurship and Business Skills Development

International Business Development

Financing Continuum

Community Development

Community Mobilization

Community-based Business Development

Community Investment

Infrastructure Programming

Policy, Advocacy and Coordination

Policy

Advocacy

Coordination

Internal Services

Governance and Management Support

Resource Management Services

Asset Management Services

Source: 2011-2012 Report on Plans and Priorities.

Appendix B: Evaluation design and methods

The mixed method evaluation design was selected in consideration of:

The CD Impact Framework suggested that case studies could be an ideal tool for considering program complexity and timeframes for achieving outcomes. Many external factors can impact the achievement of outcomes. Case studies provided an opportunity to study how impacts are achieved with this programming, while considering important contextual factors. Program theory also suggested that the achievement of intermediate outcomes could take longer than four years. Therefore, the evaluation included a follow-up, or update of results on the case studies from the previous evaluation.

The Treasury Board Policy on Evaluation (2009) requires a risk-based approach to evaluation design. That is, the evaluation approach and level of effort should be calibrated to reflect the risks associated with the programming, while meeting minimum standards of rigour related to research design and lines of evidence. The level or risk associated with CM and CI programming was considered through consultations, preliminary document review, and consideration of previous evaluation findings.

Risks associated with program relevance were considered low to medium, given materiality of the programming and changes that have occurred in the programming environment such as the continued decline of rural areas. Relevance was evaluated through document review and interviews with stakeholders and subject matter experts. Risks associated with effectiveness of the programming was considered medium to high given the materiality of the programming, information needs of management, and program complexity (i.e. complex operating environment, varied nature of activities and longer timeframe for achieving outcomes). Therefore, the assessment of effectiveness would require greater effort, using administrative data review, interviews, document review and case studies. Risks associated with program efficiency/economy were considered to be low to medium, because of the availability of basic information on program costs, which could be supplemented by other types of information. The approach to evaluating efficiency and economy included document review, interviews and data analysis.

A review of approaches to evaluating community development planning and investment programming in other jurisdictions supported the use of a mixed methods research design to answer the key evaluation questions. The following methods were used for the study:

Table B-1: Profile of case studies conducted

Project Type

Types of Case

Region

Achieved Outcomes

Lessons Learned

* symbol represents a follow up case study to the previous evaluation

Recreation/Cultural Infrastructure

Yes* Yes

NS /NB

Basic Infrastructure

Yes Yes

NS/ PEI

Marketing Industry

Yes Yes

HO/NL

Planning Industry

Yes* Yes

NB/NL

Tourism Attraction

Yes* Yes

CB/PEI

Business Skills Development

Yes*  

NL

Regional Development Planning and Mobilization - Indigenous

Yes  

CB

*These were case studies that had been conducted as part of the previous program evaluation in 2009. A follow-up was conducted to capture longer-term impacts.

The evaluation questions and judgment criteria by which outcomes would be assessed are presented below in Table B-2. These also influenced the design of the evaluation.

Table B-2: Evaluation questions, judgment criteria and methods

When judgment criteria in the table below refer to the results of previous evaluations as a benchmark for assessing current program success, it should be noted that in some cases, baseline information will not be available. In those cases, this evaluation will be used to gather baseline data for future evaluations.

Evaluation Questions

Evaluation Judgment Criteria

Method

Document Review

Interviews / Focus Groups

Surveys

Administrative Data

Case Studies

Relevance

Issue 1: Continued Need for the Programming

1.1. To what extent do the CM/ CI PSAs continue to address a demonstrable need?

The programming related needs are still present to at least the same degree as they were five years ago.

x

x

   

x

1.2 To what extent are the CM/ CI PSAs responsive to existing and emerging needs of Canadians? What other mechanisms exist to address these needs?

Community mobilization planning and investments respond to emerging needs. ACOA is aware of changing economic development needs and adjustments to programming are made to meet those needs. Roles and responsibilities of other service providers do not duplicate those of ACOA (extent of duplication, overlap or complementarity).

x

x

   

x

Issue 2: Alignment with Government Priorities

2.1 To what extent are the CM/ CI PSAs aligned with federal government priorities and expectations, and to ACOA’s strategic outcome?

There is logical alignment between the programming, federal government priorities (including evolving priorities) and ACOA’s strategic outcome. The alignment is recognized and/or made explicit.

x

x

     

Issue 3: Alignment with Federal Roles and Responsibilities

3.1 To what extent do the CM/ CI PSAs align with federal roles and responsibilities?

ACOA’s roles and responsibilities in this area of programming are aligned with the ACOA Act. Similar federal organizations in other jurisdictions administer such programming.

x

x

     

Performance

Issue 4: Effectiveness - the extent to which CM/ CI objectives have been achieved within the context of expected results and outcomes. [XXI]

4.1 How and to what extent are the CM/ CI PSAs achieving expected outcomes?

x

x

x

x

x

4.1.1 Incrementality: What impact would the absence of the CM/ CI programming have on projects/initiatives?

The impact of the absence of the programming is similar or greater to that observed in the previous evaluation, as reflected by:

  • ACOA clients’ opinion of whether the project would have proceeded without funding
  • ACOA’s influence on the involvement of funding partners
   

x

   

4.1.2 How and to what extent have CM/ CI PSAs contributed to immediate outcomes:

Planning

  • The ability/capacity to identify economic opportunities
  • Leadership skills relating to economic development planning
  • Improved collaboration between stakeholders in economic development planning
  • Regional or sector specific economic development plans and strategies

Investment

  • Development of critical economic development infrastructure, including non- physical infrastructure
  • Increased ability of communities to respond to opportunities and economic crisis
  • Partnerships/collaboration for economic development
  • Response to economic opportunities or crises.

There is sufficient evidence to argue the contribution of CM and CI programming to the achievement of immediate outcomes.

The contribution of planning and investment activities to immediate outcomes is similar or greater to that observed in the previous evaluation [XXII] as reflected by:

Planning

  • 50 per cent projects or more will result in increased community or organization capacity for strategic planning relating to economic development;
  • 50 per cent projects or more will result in increased leadership for strategic planning relating to economic development;
  • 60 per cent or more projects will result in improved collaboration relating to economic development within the community or other organizations (stakeholders);
  • 90 per cent of strategic plans have been finalized
  • Plans meet “quality” criteria that is more likely to lead to economic development outcomes (i.e. prioritization of activities and partners that are making resource commitments, proposed initiatives build on assets for economic development, etc.); and/or
  • Strategic plans identify opportunities that can lead to economic impacts such as employment/business creation, sustainability or growth.

Investment

  • The average number of partnerships/collaborators on projects is 2.7 or greater;
  • All projects identify collaborators;
  • 50 per cent projects have resulted in increased ability of communities to respond to subsequent opportunities and economic crisis;
  • Internal tools and mechanisms allow for strategic decision making in the selection of projects that result in critical economic infrastructure (or non-physical infrastructure) [XXIII] being developed; and/or
  • Projects resulted in a response to an economic opportunity or crisis.

The Agency activity targets set as part of performance measurement for CM and CI have been achieved (e.g. # of projects, # of plans, and # of partnerships).

x

x

x

x

x

4.1.3 How and to what extent have the CM and CI PSA activities and immediate outcomes contributed to the achievement of intermediate outcomes:

Planning

  • The implementation of plans and strategies that have led to economic impacts.

Investments

  • Economic activities (within businesses, communities and industries)
  • The creation of economic opportunities that have led to economic outcomes in the community.

There is sufficient evidence to argue the contribution of CM and CI programming to the achievement of intermediate outcomes.

The contribution of planning and investment activities to intermediate outcomes is reflected by:

Note: Current benchmarks for the following criteria are not available.

Planning Plans contain initiatives that have been implemented and have increased capacity for economic development, as reflected by:

  • Jobs created or maintained
  • Reduced out migration
  • Creation or survival of businesses
  • Increased sales and revenues for businesses in the region
  • Attraction of investment
  • Increased exports
  • Developed new markets
  • Diversified exports
  • Product development
  • Business skills developed
  • Tourism visitation influenced by planning activities.

Investments Investment activities yield impacts that increase capacity for economic development, such as:

  • Jobs created or maintained
  • Reduced out migration
  • Creation or survival of businesses
  • Increased sales and revenues for businesses in the region
  • Attraction of investment
  • Increased exports
  • Developed new markets
  • Diversified exports
  • Product development
  • Business skills developed
  • Tourism visitation

Note: Case studies will play an important role in understanding the type of intermediate outcomes and impacts which can be expected, and the conditions or processes involved.

x

x

x

 

x

4.1.4 What are the barriers to achieving CM and CI immediate and intermediate outcomes and to what extent are these being mitigated?

Evidence of factors that are impeding success of the CM and CI PSAs are known, and mitigation strategies are implemented where applicable.

x

x

x

 

x

4.1.5 What lessons have been learned in the implementation of CM and CI programming? How can these lessons contribute to future programming?

Not applicable

x

x

x

 

x

4.1.5 What unintended outcomes have been achieved?

Not applicable

 

x

x

 

x

4.1.6 To what extent are the CM and CI PSAs performance measurement and reporting structures effective in reporting on the achievement of outcomes? How is the performance information used by ACOA-funded organizations and by ACOA?

Evidence that CM and CI PSA performance measurement is adequate and effective in reporting on the achievement of outcomes.

Performance measurement information is used by ACOA and ACOA-funded organizations to support decision-making.

Performance information is available for use in the evaluation.

x

x

 

x

x

Issue 5: Efficiency and Economy - the extent to which CM and CI activities are undertaken in an affordable manner, taking into consideration the relationship between outputs and the resources to produce them; the extent to which resources allocated to the CM and CI PSAs are well-utilized, taking into consideration alternative delivery mechanisms.

5.1 In the context of the results being achieved, to what extent are the resources (e.g. FTEs, financial) allocated to CM and CI PSAs efficiently utilized?

ACOA has in place mechanisms to ensure that the most efficient and economical means are being used to administer the programming. For example:

  • Budget
  • Plans
  • Forecasts
  • Resource Re-allocations (manner in which decisions were made)
  • Governance/Processes
  • Timeliness of delivery

Delivery costs across regions reasonably compare to one another within the context of factors affecting regional delivery.

x

x

 

x

 

5.2 Is there a more efficient and effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned?

The organization takes into consideration its own best practices in improving alternative approaches (efficiency).

Program management has considered and continues to explore alternative modes of delivery outside the organization.

Best practices were identified and implemented.

x

x

     

 

Evaluation limitations and strategies applied to mitigate these are presented in table B-3 below.

Table B-3: Evaluation Challenges and Mitigation Strategies

Challenge

Mitigation

Shifting program priorities: During the evaluation period, the programming adapted to enable the delivery of the EAP programming and deliver on emerging Agency priorities.

Contextualized and described changes over time. Sought information on the effect of the changes on program delivery.

Limited information on planning projects: With the closure of REDOs, it was not possible to include them as evaluation participants, which limited the representation of planning projects in the client survey.

Described of funding decision, used existing performance data on REDO activities and acknowledged limitation. With regards to the interpretation of survey findings, numbers rather than percentages were reported where appropriate so as to contextualize the interpretation, acknowledging the limitation.

Program complexity: CM and CI programming aims to influence community economic development outcomes, which is affected by a broader system of variables and a wide range of stakeholders, making the study of CED complex. Also, CI programming aims to influence outcomes through a wide variety of initiatives, adding complexity to the study of program relevance and performance.

CD program theory was further developed in consultation with experts and key program stakeholders to better understand and acknowledge the context within CED outcomes are achieved. Evaluation was designed to capture complexity.

Data availability/reliability: Information on funded activities, performance and operational activities was not clear or consistently available.

To better understand funded activities, the team conducted a detailed analysis of all projects with input and validation from programs. The evaluation included methods, particularly the client survey and case studies, to gather performance information. These data are however subject to the limitations inherent to the survey and case studies methods. These include possible self-selection bias and other respondent biases and reductions in data validity due to the need to recreate data on past performance. The evaluation team developed detailed regional profile documents with input from programs to understand operations.

Difficulties in reaching participants: Interviewees could not always be reached for interviews. Similarly, survey participants could sometimes not be reached due to issues with outdated client contact information and change in personnel.

Interviews: Identified alternative interviewees Surveys: sent reminders, telephone option, email from ACOA, extended cut-off date.

 

The evaluation featured the following strengths:

Appendix C: Community economic development conceptual/analytical framework

(D)

Appendix D: Management action plan

DATE: December 13, 2013

PROJECT TITLE: Evaluation of ACOA Community Mobilization and Community Investment Sub-programs

RESPONSIBILITY CENTRE: Community Development Program Activity

RESPONSIBILITY CENTRE MANAGER: Marc Lemieux, A./Director General, Community Development

Recommendation 1: In order to ensure a clear corporate direction for programming while allowing for regional variability and flexibility, ACOA should build on its corporate knowledge, lessons learned and current best practices in CED to formulate a Community Investment framework. The framework should outline the Agency’s approach to planning and investment programming (including main project types and expected outcomes), while acknowledging regional context, plans, priorities and best practices. It should also describe the Agency’s approach to engage, coordinate and collaborate both internally and externally (partners, subject matter experts) in the achievement of programming outcomes.

Management response: Management agrees with the recommendation. Many of the elements of the framework suggested in the recommendation already exist in the Agency’s practices which will be built upon to develop the framework.

Planned actions: The Framework will describe how the Agency will focus its efforts in the development of strategic investment plans as well as in its approach to support investments in community projects that contribute to the outcomes of community investment. In addition, the framework will consider the need for regional flexibility, outline best practices and highlight the Agency’s approach for engagement with key stakeholders.

Responsibility: Head Office Director General of Community Development and Directors General Programs/Operations Committee

Target Date: Fall 2014

Recommendation 2: In support of results-based management, decision-making, and the strategic coordination of initiatives that cut across programming areas (e.g. business skills development, tourism, trade, innovation), ACOA should ensure that quality and timely information on activities (i.e. main project types), and outcomes are systematically collected on file and in electronic systems.

Management response: Management agrees with the recommendation. The performance information will support decision-making and capture learnings to enhance performance.

Responsibility: Head Office Director General of Community Development and Directors General Programs/Operations Committee

Target date: March 31, 2014

Planned actions: For projects that are coded under Community Investment, project types and their relationship to outcomes will be defined within the existing structure of the Agency’s Access system. The Community Investment Framework will make use of the information to support decision-making and facilitate linkages of projects to expected outcomes. The existing Results Tracking System found in the Agency’s Q Access system will be used to track project outcomes. Training and guidance will be provided to ACOA Program Officers on the project types, outcomes and quality of data to be inputted in the system. Monitoring on the quality of data entered in the system will be done on a regular basis.

Recommendation 3: In an effort to increase efficiency and effectiveness of programming and mitigate risks of not achieving expected outcomes, ACOA should build on current efforts to develop and systematically implement a standardized process/tool to assess project viability. The process/tool should assess the likelihood of achieving outcomes, as well as the human, financial, and infrastructure capacity of the proponent organization or community to conduct the work outlined in the proposal and to sustain the achievement of outcomes and impacts following project completion.

Management response: Management agrees with the recommendation and will build on the current practices and tools already available.

Planned actions: Develop and implement a risk assessment tool for non-commercial projects that will be used during the project assessment and documented within the existing Project Summary Form. Training and guidance will be provided to ACOA Program Officers on risk management for non- commercial projects.

Responsibility: Head Office Director General of Community Development and Directors General Programs/Operations Committee

Target: Fall 2014

Appendix E: Organizations supporting CED planning and investment in Atlantic Canada

Organization

Programming Characteristics

ACOA

  • Focus on CED outcomes for Atlantic Canada
  • Provides non-repayable funding to communities and industry sectors for: construction/infrastructure; equipment; marketing/attracting investment; skills development; events; tourism operations; immigration/recruitment of skilled labour; planning and studies
  • Manages Infrastructure Canada programming (See Other Federal Departments)

ECBC

  • Delivers CM and CI programming on Cape Breton Island as well as other funding to non-commercial organizations for planning and investment activities
  • Uses discretionary funding to support economic development initiatives; sometimes this is used to co-fund projects funded through CM and CI.

Other Federal Departments

  • Canadian Heritage (CH): Provides non-repayable funding to non-profit organizations for projects linked to Canadian history, culture, linguistic and geographic diversity such as festival; events; and planning studies for renovations and construction.
  • Infrastructure Canada (IC): On behalf of IC, ACOA provides non-repayable contributions to Provincial governments, who support infrastructure relating to water and waste, municipal environmental energy improvements; transportation; tourism; and planning studies. Decisions are not based on economic development impacts
  • Aboriginal Affairs and Northern Development Canada (AANDC): Provides non- repayable contributions for economic development organizations/services, training, capacity building, investment promotion, and research. Also provides non-repayable funds for core municipal infrastructure such as water and waste, not with an economic development lens.

Provincial Governments

Departments with economic development mandates NB-Regional Development Corporation; NL-Innovation, Business & Rural Development; NS-Economic and Rural Development and Tourism; PEI-Fisheries, Aquaculture, and Rural Development

  • Non-repayable funds for planning and implementing economic development initiatives, including infrastructures relating to recreation, municipal services, events, etc. Types of support vary by region.
  • Often co-fund CI projects with ACOA; provide less funding than ACOA generally, varies by project type and region

Other Sector-Specific Provincial Departments

  • Departments relating to tourism, culture, aquaculture and fisheries, natural resources, agriculture.
  • Funded projects align with the departments’ key sectors and are typically represent smaller funding envelopes. Less frequent co-funders of ACOA CI projects.

Municipalities

  • Provide mainly funding for the development and maintenance of municipal infrastructures relating to transportation, tourism and recreation, along with events and local economic development planning (larger municipalities often have dedicated resources).
  • Capital budgets based on tax base and municipality needs and priorities.
  • Investments vary by type of projects and capacity to leverage funds from other partners.

CBDCs

  • Locally operated corporations that offer repayable loans and business support services, mostly to loan clients.

Other

  • UBDCs in NS are located in rural areas and provide advice and knowledge for businesses
  • Provincially funded business support programs in NL target specific groups

Appendix F: Approved ACOA CM and CI projects by type and sub-type, 2008-2009 to 2011- 2012

Planning or Investment

Project Type and Project Sub-Type

Projects

ACOA Approved

Number

 per cent

$M

 per cent

Investment

Construction/infrastructure

291

26.8

112.3

49.4

community building Construction of community buildings that support the economic development of a region or community, usually rural. These include large multi-purpose centres.

34

3.1

23.3

10.3

downtown/waterfront enhancement Improving downtown public spaces, including façade, streetscape and waterfront.

21

1.9

5.9

2.6

enabling infrastructure Infrastructure needed for business or industry development /sustainability; including infrastructure for electricity; air transportation; sewer drainage; broadband; and transportation.

16

1.5

8.3

3.7

industry sector Construction in support of key industries including cruise; mining; tourism; aerospace and defence; agriculture/agri-food; arts and crafts; forestry; IT; knowledge; manufacturing; meetings and conventions; and renewable/alternative energy.

38

3.5

22.0

9.7

recreational/cultural Recreational sites that are related to tourism including nature trail/park; marina; sports centre/outdoor facility; and performing/visual arts facility.

77

7.1

16.7

7.4

pre-commercial/industrial space Pre-development of commercial space to enable business; including farmer’s market, industrial centre, retail space/restaurant, incubation centre, and business offices.

24

2.2

9.8

4.3

cultural/heritage Concert/event venue, heritage site/museums, and performance visual arts facilities.

69

6.4

21.0

9.1

training/R&D infrastructure Industry training or R&D; including marine, agriculture/agri-food, sustainable/renewable energy, oil and gas, post-secondary.

11

1.0

5.3

2.3

Investment

Acquire equipment

73

6.7

14.3

6.3

industry sector Support for industries including entertainment, transportation, fishery, manufacturing, mining, meetings and conventions, agriculture/agri-food, aquaculture, boat building, cruise, environment, and waste management.

22

2.0

5.3

2.4

recreational/cultural Support for recreational/cultural events/activities/sites; including performing/visual arts facility, broadcasting facility, festival infrastructure, multi-purpose community centres, concert venue, museum/interpretive centre, and sports facility.

36

3.3

5.7

2.5

training/R&D Support for the development of industry; including energy, manufacturing, mining, education, marine, and science and technology.

15

1.4

3.3

1.4

Investment

Event

166

15.3

13.2

5.8

business development Awards/celebrations, workshops

7

0.6

<0.1

<0.1

industry sector Hosting events related to an industry (i.e. film, mining, agriculture industries); including festivals, conferences, awards, trade shows, conference attendance, and expos.

57

5.2

1.8

0.8

policy/public administration Hosting or participating in conferences related to policy or public administration.

2

0.2

0.2

0.1

recreational/cultural Hosting or participating in recreational or cultural events; including conferences, celebrations/commemorations, and festivals

54

5.0

6.3

2.8

regional economic development Hosting events related to promoting regional CED; including conferences and tradeshows.

23

2.1

0.9

0.4

sporting/athletic Hosting tournaments, festivals, and celebrations/commemorations.

23

2.1

3.9

1.7

Investment

Immigration/recruitment

9

0.8

1.8

0.8

immigration Enhancing immigration to the region including francophone and general immigrants.

2

0.2

<0.1

<0.1

recruitment Recruiting labour: skilled workers, highly skilled workers, and international students.

7

0.6

1.7

0.8

Investment

Event Cultural or sporting events with tourism implications; including celebrations, festivals, and sporting events.

23

2.1

0.9

0.4

industry sector Promoting specific industries including: arts and crafts, meetings and conventions, fisheries, film, agricultural/agri-food, oil and gas/marine, aerospace and defence, aquaculture, boat building, music, wine, and tourism.

15

1.4

11.2

5.0

regional economic development Promoting a region for economic investments

3

0.3

0.2

0.1

services Building awareness for services including offshore safety, and training and employment.

3

0.3

0.1

<0.1

tourism attraction Attracting visitors to tourist areas or places; including region, golf experience, historic site, park/ecological reserve, museum/ interpretative centre, performing/visual arts, culinary experience, genealogy tourism, and nature trails.

74

6.8

13.6

6.0

event Cultural or sporting events with tourism implications; including celebrations, festivals, and sporting events.

23

2.1

0.9

0.4

Investment

Skills development

76

7.0

10.4

4.6

general SMEs Developing management, organizational and entrepreneurial development skills; including management skills/productivity, IT, IT management skills capacity, entrepreneurship, HR, innovation and commercialization, innovation and product development, investment readiness, land management, and supplier development.

29

2.7

4.9

2.2

industry sector Training related to specific industries including technical attendance, technical develop/deliver, management skills/productivity, HR, SROI analysis, market readiness, tourism, and productivity.

43

4.0

4.8

2.1

regional economic development Skills development related to community economic development.

4

0.4

0.6

0.3

Investment

Tourism attraction operation

11

1.0

1.6

0.7

historic site Operational funding for historic sites.

5

0.5

0.8

0.4

performing arts Operational funding for performing arts.

5

0.5

0.7

0.3

museum/interpretive centre Operational funding for museum/interpretive centres.

1

0.1

0.1

0.1

Investment

Research centre operation

5

0.5

4.6

2.0

industry sector Operation funding for industry sectors; including fisheries, mining, and ocean technology

5

0.5

4.6

2.0

Planning

Planning

338

31.1

42.6

18.8

industry sector Planning for the development of an industry overall; including tourism, oil and gas, mining, forestry, agriculture/agri-food, arts and crafts, fishery, music, transportation, aerospace and defence, forestry, harness racing, IT, knowledge, manufacturing, music, performing arts, renewable energy, and wine.

69

6.4

7.4

3.3

project specific Planning for a particular initiative; including recreational/cultural infrastructure, recreational/cultural attractions, recreational/cultural events, port/harbour infrastructure, enabling infrastructure, downtown/waterfront infrastructure, business development, community buildings, academic, and regional tourism signage.

116

10.7

6.2

2.7

regional development Planning for community economic development including rural, urban, Indigenous, francophone, northern, and black business.

65

6.0

9.2

4.1

TOTAL

1087

100

226.96

100

Appendix G: CM and CI projects by main types and regional office, 2008-09 to 2011- 2012

Project Type

NS

No. Projects

Approved Funding

$(M)

 per cent

Construction/ Infrastructure

81

24.7

52.4

Acquire equipment

7

0.8

1.7

Event

8

3.4

7.0

Immigration/ recruitment

4

0.5

1.1

Marketing

15

1.8

3.9

Skills development

4

3.4

7.3

Tourism site operation

     

Research Centre Operation

     

Planning

85

12.5

26.5

Total

204

47.1

100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type

CB

No. Projects

Approved Funding

$(M)

 per cent

Construction/ Infrastructure

47

16.8

55.4

Acquire equipment

23

3.1

10.3

Event

19

2.0

6.6

Immigration/ recruitment

 

��

 

Marketing

19

3.9

12.8

Skills development

6

0.7

2.2

Tourism site operation

1

<0.1

0.1

Research Centre Operation

1

0.5

1.6

Planning

42

3.3

11.0

Total

158

30.4

100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type

NL

No. Projects

Approved Funding

$(M)

 per cent

Construction/ Infrastructure

77

29.8

46.3

Acquire equipment

22

4.1

6.4

Event

88

3.5

5.5

Immigration/ recruitment

1

<0.1

<0.1

Marketing

44

3.7

5.7

Skills development

27

3.5

5.4

Tourism site operation

5

0.9

1.4

Research Centre Operation

3

2.5

3.9

Planning

118

16.3

25.3

Total

385

64.4

100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type

PEI

No. Projects

Approved Funding

$(M)

 per cent

Construction/ Infrastructure

56

12.5

29.8

Acquire equipment

12

1.3

3.1

Event

30

2.7

6.3

Immigration/ recruitment

     

Marketing

34

16.1

38.4

Skills development

5

0.9

2.1

Tourism site operation

4

0.6

1.5

Research Centre Operation

     

Planning

58

7.9

18.9

Total

199

42.0

100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type

HO

No. Projects

Approved Funding

$(M)

 per cent

Construction/ Infrastructure

     

Acquire equipment

     

Event

4

0.1

6.7

Immigration/ recruitment

1

<0.1

1.3

Marketing

1

0.3

12

Skills development

5

0.6

32.7

Tourism site operation

     

Research Centre Operation

     

Planning

9

0.9

47.3

Total

20

1.9

100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Project Type

NB

No. Projects

Approved Funding

$(M)

 per cent

Construction/ Infrastructure

30

28.5

69.1

Acquire equipment

9

5.0

12.1

Event

17

1.6

4.0

Immigration/ recruitment

3

1.2

2.9

Marketing

5

0.3

0.6

Skills development

29

1.3

3.2

Tourism site operation

1

0.1

0.2

Research Centre Operation

1

1.6

3.9

Planning

26

1.7

4.2

Total

121

41.2

100

Source of project numbers and approval amounts: QAccess.

Source of project types: coding scheme developed by evaluation team in consultation with the CMCI evaluation working group. EAP projects are not included.

Appendix H: Annual salary costs per region

Table H-1: community mobilization and community investment salary per region by fiscal year

Region

Salary ($)

2008-09

2009-10

2010-11

2011-12

Total ($)

N.L.

1,496,248

1,829,430

1,853,406

1,866,929

7,046,013

N.S.

1,156,315

1,445,546

1,218,942

1,063,134

4,883,937

N.B.

1,934,500

1,683,814

1,154,064

1,325,458

6,097,837

P.E.I.

204,990

545,207

656,953

599,752

2,006,902

H.O.

299,824

810,133

706,646

672,755

2,489,358

C.B.

473,602

582,154

585,203

519,913

2,160,872

Total

5,565,479

6,896,284

6,175,214

6,047,941

24,684,919

Source: GX Financial Data, August 2012

Appendix I: End notes

[1] Government Organization Act, Atlantic Canada, 1987, Part 1, R.S., c G-5-7.

[2] TB Policy Suite on Evaluation: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx? section=text&id=15024

[3] Patton, M.Q. (2008) Utilization-Focused Evaluation, 4th Edition

[4] Europeaid Cooperation office, extracted May 2012. 

[5] ACOA. Innovative Communities Fund Terms and Conditions. 2011.

[6] ACOA. Business Development Program Terms and Conditions.

[7] ACOA. Evaluation of Community Investment and Community Development Resources Program Sub-Activities. 2009.

[8] ACOA Departmental Performance Report 2010-2011 

[9] Chen, H.T. (1990). Theory-driven evaluations. Sage Publications.

[10] Ibid.

[11] TBS (2012). Theory-Based Approaches to Evaluation: Concepts and Practices. http://www.tbs-sct.gc.ca/cee/tbae-aeat/tbae-aeat- eng.pdf

[12] ACOA Departmental Performance Report 2011 - Atlantic Canada’s 2010-2011 Economic Overview and Provincial Economic Profiles.

[13] ACOA, Community Development Impact Model, p.8.

[14] Vasseur, L. & Catto, N. R. (2008). Atlantic Canada; in From Impacts to adaptation: Canada in a changing climate 2007, edited by Lemmen, D. S., Warren, F. J., Lacroix, J. & Bush, E. Government of Canada, Ottawa, ON, p. 119-170.

[15] ACOA, Community Economic Development Impact Framework, (March, 2012).

[16] ACOA, Community Investment and Community Development Resources Evaluation, (Dec. 2009).

[17] Royal Bank of Canada Provincial Outlook 2013

[18] Community Investment and Community Development Resources Evaluation, (Dec. 2009).

[19] APEC: Meeting the skills challenge, (2012).

[20] APEC: Meeting the skills challenge, (2012).

[21] Bourgeois, Surette & Thériault, The knowledge economy in Atlantic Canada. Centre for Innovation and Productivity, (2011).

[22] APEC: Meeting the skills challenge, (2012).

[23] Statistics Canada Labour Force Survey

[24] Roger Martin and James B. Milway, Institute for Competitiveness and Prosperity, Strengthening Management for Prosperity (May 2007), page 18.

[25] Penelope M. Rowe, The Non-Profit and Voluntary Sector in Atlantic Canada: Regional Highlights of the National Survey on Non-Profit and Voluntary Organizations, (2006).

[26] Richard Matoon, Paper on the role of infrastructure for economic development, Economic Conference, Ottawa Canada (2004).

[27] Statistics Canada, Age of Public Infrastructure: A Provincial Perspective written by Michele Gagnon, Valerie Gaudreault and Donald Overton, (2008).

[28] Penelope M. Rowe, The Non-Profit and Voluntary Sector in Atlantic Canada: Regional Highlights of the National Survey on Non-Profit and Voluntary Organizations, (2006).

[29] Conference Board of Canada, Canada Enters a Period of Fiscal Restraint, ( March 31, 2011): http://www.conferenceboard.ca/press/newsrelease/11-03- 31/canada_enters_a_period_of_government_fiscal_restraint.aspx

[30] Goss Gilroy Inc. Summative Evaluation of the Economic Development Initiative, under the 2008-2009 Strategy for Official Languages: Roadmap for Canada’s Linguistic Duality. (July, 2012)

[31] Mount Allison University, Baseline Data for Aboriginal Economic Development: An Informed Approach for Measuring Progress and Success, (Nov. 2010).

[32] ACOA, Program Alignment Architecture, 2011-2012.

[33] Treasury Board Secretariat of Canada, Whole-of-government framework, I Economic Affairs: 

[34] 2008-2011, 2012 Federal Budget Speeches

[35] Speech From the Throne 2008, 2009; Budget 2012

[36] Speech From the Throne 2008

[37] Speech From the Throne 2010-2011 and Budget 2012

[38] Speech From the Throne 2010

[39] ACOA Act, (R.S.C., 1985, c. 41 (4th Supp.) http://laws- lois.justice.gc.ca/eng/acts/A-13.7/page-1.html

[40] Idem.

[41] ACOA DPRs, 2008-9 to 20011-12:

[42] Goss Gilroy. Summative Evaluation of the Economic Development Initiative Under A Roadmap for Canada’s Linguistic Duality 2008-2013. 2012

[43] ACOA. Évaluation du programme sur l’immigration francophone (Phase 1). 2012

Footnotes

[I] In order to properly contextualize findings, some references have been made to activities occurring outside the scope of this evaluation.

[II] Due to challenges associated with segregating costs of delivering initiatives associated with EAP and Roadmap for Linguistic Duality, the evaluation of efficiency and economy included these as part of all programming delivered under CI.

[III] Two case studies were completed by Performance Management Network, and the client survey was administered by Goss Gilroy Inc.

[IV] The overall number of projects was 1,087, and these were conducted by 556 client organizations. After removing REDOs, organizations whose projects had just begun (due to their inability to report on outcomes), and organizations whose contact information was invalid, the number of individual clients surveyed was 465.

[V] These impacts were identified in consultation with program managers during the evaluation planning phase and development of the CED Conceptual/Analytical Framework

[VI] Due to the complexity of the programming, the evaluation unit expanded upon the logic models to develop a broader program theory, in consultation with program and subject-matter experts. The CED Conceptual/Analytical Framework outlines the relationship between ACOA CD programming activities and their interaction.

[VII] Economic challenges abroad such as failure of the US economy to make gains, the European recession and slowdown in Chinese economic growth, fostered a challenging environment for economic recovery in Canada overall.

[VIII] ACOA. Evaluation of the CI, CDR, OLMC and Aboriginal Communities Program Sub-Activities, 2009

[IX] It should be noted that outcome measurement for initiatives that resulted from planning would have likely benefited from a longer measurement time frame.

[X] As some regions adopted the practice of distributing new funding through amendments to REDO projects, approval amounts are lower than actual expenditures. Actual expenditures for REDO operational activities were $29.6 million over the four-year period.

[XI] These results represent the number and percentage of completed projects for which proponents reported the achievement of outcomes to either some extent or a great extent.

[XII] These results represent the number and percentage of completed projects for which proponents reported the achievement of outcomes to either some extent or a great extent.

[XIII] These percentages are based on the number of projects for which proponents reported the achievement of outcomes to either some extent or a great extent.

[XIV] This time frame was most relevant for studying linkages to ACOA’s Strategy for Tourism Growth, as it was implemented by early 2009.

[XV] The analysis of projects in support of tourism was supported by ACOA’s Tourism Atlantic group and further validated by regional CD managers and directors.

[XVI] The survey asked proponents to report the number of other organizations involved in their projects.

[XVII] DPR targets for the number of collaborators per project were largely met and ranged from 650 collaborators in 2009-2010 to 324 in 2011-2012.

[XVIII] Due to ACOA’s decision to end core operational funding under the current REDO model effective May 2013, these organizations will be excluded from interviews.

[XIX] Due to ACOA’s decision to withdraw end core operational funding under the current REDO model effective May 2013, these organizations will be excluded from the client survey.

[XX] ACOA (2012). Community Economic Development Conceptual/Analytical Framework. Draft Project Report.

[XXI] Generally, the wording of expected outcomes is derived from the logic models. However, as mentioned in Section 2.1, it was found in consultations that the original logic models for CM and CI did not fully capture desired outcomes and changes were suggested. Wording of expected outcomes has been modified to reflect the consultation findings.

[XXII] The targets (%) were identified based on previous evaluation findings.

[XXIII]In the 2009 evaluation of CI/CDR, ACOA adopted an OECD definition of critical infrastructure for economic development: Canada’s critical infrastructure consists of those physical and information technology facilities, networks, services and assets which if disputed or destroyed, would have a serious impact on the economic well-being of Atlantic Canadians.

 

Page details

Date modified: