This evaluation provides the Atlantic Canada Opportunities Agency’s (ACOA) management with objective, neutral evidence on the relevance and performance of its Productivity and Growth (P&G) sub-program. The results of this evaluation will be used to support ongoing program management and future direction of the programming.

The study was managed and implemented by ACOA’s Evaluation Unit. We thank the members of the evaluation advisory committee (EAC) for their advice and support throughout the evaluation process. These included ACOA’s director general of Enterprise Development, P&G directors across all ACOA regions, a subject matter expert from within ACOA’s policy function, external subject matter expert Eric Cook of the N.B. Research and Productivity Council, and external evaluation analyst Marilyn Cree, of Western Economic Diversification. Their involvement and assistance helped to ensure that the study and its recommendations are both relevant and useful. The evaluation team was also supported by the advisory services of economist Dr. Andrew Sharpe, Centre for the Study of Living Standards, who provided subject matter expertise outside of the EAC.

We are also grateful to all others who provided their time and essential knowledge in support of the study such as the many ACOA staff members, members of the Evaluation Unit, and external key informants (KIs).

Overall, these contributions were instrumental in building our understanding and identifying best practices and areas for improvement for ACOA’s Productivity and Growth sub-program.

Courtney Amo
Director, Evaluation and Risk Directorate (Head of Evaluation)
Atlantic Canada Opportunities Agency

Executive Summary

This report presents the findings, conclusions and recommendations of the evaluation of ACOA’s Productivity and Growth (P&G) sub-program. The evaluation examines the relevance and performance of P&G programming from 2008-2009 to 2013-2014 as per the Agency’s approved evaluation plan and the evaluation’s terms of reference. It meets the accountability requirements outlined in the Treasury Board’s (TB) Policy on Evaluation.[i]

The P&G sub-program supports Atlantic Canadian small and medium-sized enterprises (SMEs) in accessing the information, business skills and financing they require to start and/or grow a business. P&G investments are made directly by helping businesses increase competitiveness through activities that support productivity improvements (e.g. the purchase of new technologies/equipment), start-ups/establishments, expansions, modernizations, domestic marketing, business skills development, training, and hiring of skilled personnel.

Indirectly, contributions are made to non-profit organizations such as business and economic development associations that undertake activities that support business productivity, business establishment, entrepreneurship and growth as well as business skills development.

Relevance

The findings indicate the P&G sub-program remains relevant. There is a continued need for programming to improve the productivity and growth of Atlantic SMEs, through access to capital, improvements in productivity and innovation, entrepreneurship promotion and support, and the development of a variety of business skills.

Key informants identified opportunities for greater collaboration and consultation with external stakeholders and experts to discuss broader policy issues, and to gain access to research in support of decision-making, including policy research for modifying programs, setting strategy, and strategic intelligence for working with clients.

Atlantic Canada’s aging population, which is the oldest in Canada, combined with the outmigration of youth have exacerbated the need for youth entrepreneurship promotion and supports, as well as the need for business succession planning to increase the sustainability of viable businesses as owners retire. As the region faces challenges with labour force participation, including self-employment, there is also a continued need to support entrepreneurship for under-represented groups such as women. During the evaluation period, entrepreneurship and business skills funding programs aimed at youth and women were folded into the Agency’s larger Business Development Program (BDP). ACOA continued to support initiatives aimed at women, and saw some decline in youth entrepreneurship programming under P&G programming. Internal key informants expressed a desire to increase the Agency’s focus on entrepreneurship, as well as coordination and information sharing across regions, which has declined in recent years. The Agency has recently taken steps to explore the further use of Gender-Based Analysis Plus (GBA+) within the Agency.

As a complement to annual planning exercises, program stakeholders raised the need for an articulated and communicated overarching, longer-term strategy for P&G that reflects key objectives, activities and priorities, covering commercial and non-commercial entrepreneurship and business skills supports (e.g. general and/or aimed at specific groups such as youth, women, sector associations, etc.).

P&G programming is aligned with ACOA’s strategic outcome and the ACOA Act. The activities are aligned with ACOA’s roles and responsibilities, as well as with broader federal priorities related to promoting small businesses and tourism, as well as creating jobs and opportunities for Canadians. The unique nature of P&G’s programming sets it apart from other funders, meaning it complements rather than duplicates other programming. However, P&G programming is also part of a complex landscape of programs offered in the region, which can be a challenge for both funders and entrepreneurs in terms of awareness and understanding.

Effectiveness: Achievement of Expected Outcomes

Overall, the evaluation found that the Agency’s P&G activities have had a positive incremental impact on the implementation of projects, leveraging of funds, and achievement of outcomes. The achievement of P&G outcomes was supported by an increase in repayable contributions approved over time. Areas of outcomes achievement for both repayable contributions and Productivity and Business Skills Initiative (PBSI) clients include increased production volume, improvements in product quality and productivity, competitive position, and employment. Notably, PBSI results also include the development of management and marketing skills. Areas for continued development and focus include productivity and the development of new markets.

ACOA-supported SMEs out-performed unassisted SMEs in sales and labour productivity growth and in business survival rates. The majority of repayable contribution funds (84%) due over this time frame were repaid to the Agency.

Outcomes achieved by non-commercial organizations supported under P&G include: increasing knowledge and skills to start or grow a business, increasing SMEs’ market knowledge, marketing skills and activities, and strategic partnerships and growth. It was suggested that ACOA continue to develop outcomes related to entrepreneurship.

Several factors facilitate success such as providing clients with timely access to expert advice. Best practices include lean productivity initiatives. Pro-active promotion of ACOA programming to SMEs by ACOA staff facilitates program delivery. Clients highly value the non-financial supports (e.g. advice/guidance) they receive. Barriers to program success, including client skills and knowledge gaps and internal constraints for accessing specialized knowledge development, are mitigated to some extent.

Efficiency and Economy

Overall, the evaluation found that P&G programming demonstrates efficiency and economy in the utilization of resources.[ii] The cost of delivering P&G programming has decreased since the previous evaluation and is reasonable compared to the delivery costs of other Enterprise Development (ED) sub-programs.

A number of mechanisms are in place related to design and delivery, which contribute to the effective and efficient use of resources, including decentralized delivery and flexible programming that is able to respond to needs at a regional level, focus in key sectors to maximize impacts, and increased reach of SMEs through not-for-profits resource leveraging, performance measurement, established governance structures, internal communication/coordination mechanisms, as well as consideration of alternative approaches to delivery.

Some existing and potential inefficiencies pertain to specific BDP policies and procedures that impact other programming areas beyond P&G. Particularly those related to funding limits and some eligibility requirements. Areas where efficiency and economy could be enhanced are through improvements to policies and procedures related to program delivery, communication and coordination on a pan-Atlantic basis and with external stakeholders, as weaknesses in these areas pose potential risks to the efficient and economical use of resources and achievement of outcomes. In terms of capacity building, staff would benefit from increased access to specialized knowledge and skills to be able to provide the advice necessary to efficiently address the growing number of complex SME issues.

Recommendations

The evaluation identifies two recommendations related to developing a long term strategic vision for P&G programming through the development of a P&G framework and enhancements in the efficient utilization of resources by improving upon processes and procedures impacting program delivery.

Recommendation 1: In light of the evolving demographics of entrepreneurs, and the increasingly complex and global nature of business, it is timely for ACOA to build on its current policy research and knowledge base, best practices, and other planning exercises to develop and communicate an overall long-term Productivity and Growth Framework that will promote a clear corporate direction for programming, while allowing for regional flexibility.

Given the programming’s decentralized delivery and regional differences in context and needs, it is recommended that the framework be developed collaboratively, updated regularly, and promoted as a strategic planning and internal communication tool, as a complement to the annual planning exercises.

The following is a list of considerations in developing the framework that build on evaluation findings:

The positioning of P&G among other ACOA supports and an increasingly complex landscape of external programming;
Key activities and initiatives, including P&G repayable contributions to certain sectors, PBSI and non-commercial programming as it relates to entrepreneurship and business skills development for certain sectors or groups (Note: identification of key activities and desired outcomes can build upon an ongoing logic model/performance measurement review and a planned Entrepreneurship and Business Skills Review);
Regional P&G context and priorities and their key role in delivery (e.g. key sectors / areas of focus for commercial and non-commercial supports);
Effective consultation and collaboration approaches with key external pan-Atlantic and regional stakeholders (collaborators, clients, beneficiaries, and others) as well as collaborators outside the region, which support ongoing relevance and success of P&G; and
Governance and other mechanisms that support internal planning, communication and coordination within and between regions, including sharing best practices and capitalizing on available information to inform programs.

Recommendation 2: Building on the efficiency and effectiveness of P&G programming, it is recommended that ACOA explore and pursue opportunities for improvements related to delivery processes, recognizing that these have implications that reach beyond P&G programming. Existing governance structures should be used to prioritize actions in the context of available resources, existing plans and priorities, as well as relative risks to efficiency and effectiveness.

In keeping with the findings of this evaluation, it is advised that management explore the following:

Review eligible sectors supported by the BDP to determine whether changes are required to reflect the evolving business realities and needs;
Aim to strike a balance between efficiency and oversight/control in adopting a risk based approach to project delivery processes, including streamlining Project Summary Form content and reviewing approval processes for large projects (>$500,000);
Review PBSI guidelines related to the total funding available over the SME’s lifespan and to its use for start-up SMEs;
Continue to build account manager knowledge and capabilities to support them in navigating the increasing complexity of business growth and business supports. This will help assist with continued strong support to SMEs and ensure stewardship of public funds.

The management action plan (MAP) prepared by P&G management to address each of the evaluation’s recommendations is presented in Appendix A. The full evaluation question framework, including judgment criteria, methodology and level of effort, is presented in Appendix B.

1.0 Introduction

This report presents the findings, conclusions and recommendations of the evaluation of the Agency’s P&G sub-program, which aims to enhance Atlantic Canadian SMEs productivity and growth capacity. These programming activities were last evaluated as part of the former Innovation,[iii] Financing Continuum,[iv] and Entrepreneurship and Business Skills Development[v] sub-activities evaluations, approved between 2009-2010 and 2010-2011.

This evaluation examines the relevance and performance of P&G programming based on the Agency’s approved evaluation plan and the evaluation’s approved terms of reference.[vi] The evaluation meets the accountability requirements outlined in the TB Policy on Evaluation[vii] and addresses specific information needs of senior management. The scope of the evaluation included P&G activities funded between 2008-2009 and 2013-2014.

All activities related to Canada Business were excluded from the scope as they were recently evaluated as part of the national evaluation of the Canada Business Network led by Industry Canada and approved June, 2014[viii]. The scope also excluded programming related to Canada’s Economic Action Plan (EAP) initiative, the Community Adjustment Fund (CAF), which due to the short-term nature of the funding, underwent a separate reporting exercise, and was not subject to formal evaluation[ix].

The evaluation approach was calibrated to the extent possible, building upon existing data and focussing efforts on issues of greatest importance to meet the information needs of senior management for timely evaluation results. Emphasis was placed on ensuring knowledge transfer from other recent evaluations Enterprise Development (ED) Program evaluations (such as Innovation and Commercialization (IC) and International Business Development (IBD) and on ensuring timeliness and utilization of evaluation results.

The evaluation was led by an evaluation team from ACOA’s Evaluation and Risk Directorate. The services of a consulting firm were used to assist with the conduct and administration of the on-line client survey. An evaluation advisory committee (EAC) chaired by the Director, Evaluation and Risk (Head of Evaluation) supported the evaluation. The committee included representation from ACOA as well as an external subject matter expert and an external evaluation expert.

Sections 1 and 2 of this report provide an overview of the evaluation approach and a profile of the P&G sub-program. Sections 3 to 5 present the evaluation study’s findings by broad area of relevance, performance effectiveness, and performance efficiency and economy. Section 6 outlines the conclusions and recommendations. The management action plan (MAP) prepared by P&G management to address each of the evaluation’s recommendations is presented in Appendix A.

1.1 Evaluation Overview

This evaluation provides timely, credible and neutral information on the relevance and performance of P&G programming to support decision making, continuous improvement and results-based management. In addition to the TB Policy on Evaluation requirements, the planning and calibration of the study were influenced by an extensive ED sub-program evaluation planning study[x] and the recently completed evaluations of the IC and IBD sub-programs.

An evaluation framework (Appendix B) was developed to address specific questions related to P&G, identified through consultations with ACOA senior management. The framework also reflects coverage of the five core issues as required by the TB Directive on Evaluation, and identifies corresponding judgment criteria and methods. The five core areas include:

Issue 1: Continued Need for the Program
Issue 2: Alignment with Government Priorities
Issue 3: Alignment with Federal Roles and Responsibilities
Issue 4: Achievement of Expected Outcomes
Issue 5: Demonstration of Efficiency and Economy

An evaluation judgment criterion or “benchmark” specifies an aspect of the evaluated intervention that will allow its merits or success to be assessed. Judgment criteria were used in this evaluation to avoid subjectivity and to formulate judgments on accepted terms, to improve the transparency of the evaluation by making judgments explicit, and to determine the required indicators, data and analysis. The judgment criteria were established with advice from program management.

1.2 Evaluation Design and Methodology

A risk-based approach was used for the design of the P&G evaluation. The evaluation was calibrated to meet the information and timing needs of ACOA management. Time and other resources were saved by maximizing the use of information generated from the extensive ED planning study and of information and tools from recent IC and IBD evaluations. Where possible, interviews were conducted in groups for greater efficiency, and focused on filling information gaps, while ensuring a proper representation of internal and external stakeholder views (e.g. other funding organizations, provincial government stakeholders and subject matter experts).

The P&G evaluation design was informed by discussions on program theory with management, and incorporated multiple methods to better understand the complex factors that influence relevance and the achievement of outcomes. The evaluation team was supported by the advisory services of economist Dr. Andrew Sharpe, Centre for the Study of Living Standards, who was contracted to provide subject matter expertise on evaluation design, methods, analysis and interpretation of findings. The survey design also benefited from the advice of a consulting firm – Ference Weicker, Inc. - with expertise in evaluating economic development programming including other ACOA sub-programs.

The evaluation used a mixed-methods research design involving multiple lines of evidence gathered through both quantitative and qualitative methods. A mixed-methods approach allowed for triangulation (i.e. convergence of results across lines of evidence) and complementarity (i.e., developing better understanding by exploring different aspects of an evaluation issue). Each method was developed in consultation with EAC methods champions prior to implementation, to maximize reliability and validity.

The following methods were used to evaluate the P&G sub-program. Details of each method are in Appendix C.

Document and literature review
Data review (e.g. ACOA’s administrative, financial and performance measurement data), and analysis of BDP business support project descriptions related to specific groups (e.g. women, youth, industry sectors, aboriginal groups, etc.)
Comparative analysis of performance between ACOA-supported firms and non-clients using data obtained from Statistics Canada
Key informant (KI) interviews
Client surveys

1.3 Evaluation Strengths, Limitations and Mitigating Measures

Evaluation strengths include the opportunity to build on knowledge gained through completed ED evaluations and previous evaluations of similar programming; the diversity of methods used, including a comparative analysis of the performance of ACOA-assisted SMEs against non-assisted SMEs in the same sectors; and the participation of program stakeholders and external experts in the governance, development of methods, and validation of findings. Evaluation limitations included short timelines for completion; complexity associated with establishing ACOA’s contribution to client outcomes; administrative coding issues; changes in the program alignment architecture (PAA) over the scope of the evaluation; challenges of establishing a SME comparison group; limited performance and project outcome data; and potential for selection bias. Mitigation measures were implemented where possible to ensure that the evaluation findings could be used with confidence to guide program planning and decision making. Detailed limitations and mitigating measures are in Appendix D.

The convergence of findings from the diversity of methods used helped compensate for the limitations of any single method or data source. Given mitigation measures, the evaluation team is confident in the validity and reliability of the findings and where applicable, has provided caveats or contextualization to support fair and accurate interpretation.

2.0 Profile of the Productivity and Growth Sub-program

With a focus on enhancing productivity and growth capacity in Atlantic Canada, the P&G sub-program resides within the Enterprise Development program in the Agency’s PAA (Appendix E), which targets “improved growth and competitiveness of Atlantic small and medium-sized enterprises.” The two other sub-programs supporting the ED program are the IBD, which aims to increase economic benefits for Atlantic Canada from international markets, and IC, which seeks to strengthen innovation and commercialization capacity in Atlantic Canada.

The ED program works in collaboration with three other program areas that comprise ACOA’s PAA: Community Development (CD); Policy, Advocacy and Coordination (PAC) and Internal Services. Combined, these four components of the PAA support the Agency’s strategic outcome of developing a competitive Atlantic Canadian economy.

The P&G sub-program supports Atlantic Canadian SMEs in accessing the information, business skills and financing they require to start and/or grow a business. Funding for P&G is provided by ACOA’s BDP, and includes the following initiatives:

Community Adjustment Fund (CAF) – 2009-2010 and 2010-2011
Women in Business Initiative (WBI) – 2002-2003 to 2009-2010[xi]
Young Entrepreneur Development Initiative (YEDI) – 2002-2003 to 2009-2010

2.1 Program Theory

A program’s theory serves to communicate the assumed causal connections between program elements. This theory is often presented visually in the form of a logic model, identifying the reach, activities and immediate and longer-term (or strategic) outcomes. Though ACOA’s P&G logic model is currently undergoing revisions, activities and expected outcomes are described below. The most recently approved logic model and a complete description of outcomes is presented in Appendix F.

Activities

P&G investments are made by providing direct repayable contributions to businesses that increase their competitiveness through activities that support productivity improvements (e.g. purchase of new technologies/equipment), start-up/establishments, expansions, modernizations, domestic marketing, business skills development, training, and hiring skilled personnel. Non-repayable contributions are also made to non-profit organizations such as business and economic development associations that support business establishment, entrepreneurship and growth as well as business skills development (See Section 2.5 for a breakdown of project types). P&G programming also includes activities supported under a project type called the Productivity and Business Skills Initiative (PBSI)[xii]. These projects, the vast majority of which are non-repayable, are targeted at improving competitiveness through training, skills development, and the development and implementation of plans for improved production efficiency, product quality or environmental management effectiveness. The Agency also administers Canada Business, which acts as a comprehensive source of information on government services, programs, regulations and resources for businesses. By reducing the complexity of dealing with various levels of government, Canada Business plays a role in the business development process.

Outcomes

Through its direct support to businesses, the P&G sub-program is expected to result in increased expansions (as indicated by increased production capability - new products or increased volume), increased modernizations (i.e. increases in efficiency of production/productivity), successful implementation of newly acquired technology/equipment, improved processes and products, enhanced business skills capacity within SMEs or increased domestic marketing capabilities and activities.

Immediate outcomes of supports to business, provided by a variety of non-governmental organizations and information services delivered through Canada Business, are expected to enhance knowledge or skills of aspiring and existing entrepreneurs in Atlantic Canada, which in turn, can support their ability to establish or grow their business, improve their productivity, market their products, etc.

These P&G immediate outcomes, over a period of time, are expected to contribute to the establishment and growth of businesses, increased sales as well as increased productivity which can contribute to increased profit margins over time. Ultimately, these outcomes are expected to contribute to “enhanced Atlantic Canadian SMEs productivity and growth capacity.”

In addition to the P&G logic model, an impact conceptual framework (Appendix G) was drafted in 2012 as part of the evaluation planning process for all ED sub-programs. The framework illustrates the types of activities undertaken, the stakeholders involved, the immediate and intermediate outcomes, and the desired impacts. It shows the complex environment in which P&G programming operates, including both ACOA’s contribution via other sub-programs and external contextual factors. The framework shows the interrelationships between P&G activities, outcomes and impacts.

2.2 Program Accountability and Governance

ACOA’s executive committee is the most senior internal governing forum that supports the president (ACOA’s deputy minister) in the development of the policies and programs of the Agency. Members include ACOA regional vice-presidents and related senior executives, as identified by the President, to ensure the appropriate representation of expertise.

Responsibility for the ED program activity and related sub-programs, including P&G, includes oversight by ACOA’s head office (HO) located in Moncton, NB, and program management by the Agency’s regional offices in Halifax, Charlottetown, St. John’s, and Fredericton. This includes the director general, ED at HO and the regional directors of ED. The director general, ED reports directly to the vice-president, Policy and Programs, while regional directors of ED report to the director generals of Operations located within their respective regions. Each regional office, led by a vice-president, is responsible for contributing to the development of ACOA policies and the delivery of Agency programs within their respective provinces.[xiii]

2.3 Project Approval and Signing Authority

ACOA has established a financial signing authority delegation instrument that stipulates signing authorities for various spending amounts, in accordance with the Transfer Payments Policy, the Policy on Delegation of Authorities, and any other applicable policies, acts or regulations.

The Business Development Program is the main transfer payment program used to fund projects under the P&G sub-program. BDP signing authority is delegated to various levels depending on the office and type of project. ACOA regions may further restrict signing authority levels according to the amount of funding and type of project. According to ACOA’s program delegation instrument, BDP managers and program officers may have the authority to approve contributions up to a maximum of $300,000. Directors, director generals of Operations and vice-presidents have the authority to approve contributions up to $500,000. ACOA’s president can approve contributions up to $1,000,000; however, direction from the minister/minister of State is to be sought for all contributions between $1,000,000 and $10,000,000.

2.4 Expenditure Profile

The Agency’s P&G sub-program activities are funded via the BDP. The expected outcomes of P&G programming support the objectives of the BDP, which are to:

improve the growth and competitiveness of Atlantic SMEs;
provide for dynamic and sustainable communities in Atlantic Canada; and
provide for policies and programs that strengthen the Atlantic economy.

Approved Funding

Table 1 provides an overview of the approved P&G projects and financial assistance in grants and contributions (G&Cs) over the six-year period covered by the evaluation.[xiv] Overall, the number of new projects and amount of funding approved has steadily increased throughout the period.

Table 1: P&G Sub-Program Projects and G&C ACOA Assistance Approved, 2009-2014[xv]

Approved Fiscal Year	Projects (#)	ACOA Assistance ($M)	Total Project Costs ($M)
2008-2009	414	45.3	107.2
2009-2010	444	44.7	115.9
2010-2011	430	53.7	150.6
2011-2012	485	57.4	174.3
2012-2013	493	69.4	191.8
2013-2014	461	61.1	171.9
Total	2,727	331.6	911.8

Source: QAccess database, May 2014

Expenditures

The P&G sub-program expenditures from 2008-2009 to 2013-2014 are outlined in Table 2. G&C expenditures provided under P&G ($277.9 million) represent approximately 29.5% of the G&C expenditures for ED over the period ($941.8 million). P&G operations and maintenance (O&M) expenditures totalled $57.4 million representing 17.1% of total P&G expenditures.[xvi]

Table 2: P&G Sub-program Expenditures, 2009-2014[xvii]

Fiscal year	G&C ($M)	O&M ($M)		Total ($M) [xviii]
Fiscal year	G&C ($M)	Salaries	General Operating	Total ($M) [xviii]
2008-2009	40.1	9.8	3.8	53.7
2009-2010	39.5	6.1	2.9	48.6
2010-2011	39.1	6.7	2.4	48.2
2011-2012	41.5	7.0	1.9	50.3
2012-2013	57.0	6.1	1.6	64.7
2013-2014	60.7	7.3	1.7	69.7
Total	277.9	43.1	14.3	335.3

Source: GX data, May 2014

2.5 Program Priorities and Project Profile

Program Priorities

The following contextual factors and priorities are important considerations related to P&G programming during the evaluation period. These were reflected in discussions with management and in some of the Agency’s annual operational and corporate plans developed throughout the period.

P&G activities were focused on meeting the Agency priority of “increasing productivity and competitiveness of Atlantic SMEs.”
Management increased focus on supporting growing businesses, accompanied by increases in targets for BDP commercial repayable contributions.
A priority was established for using PBSI to fill skills gaps with SME clients.
The decision was made to not renew specific funding initiatives that targeted entrepreneurship and business skills development activities for women and youth, in favour of continuing to support these types of projects within the broader BDP.
Regional plans reflected focus in key sectors.

Project and Client Types

Table 3 shows the main types of P&G products supported in Atlantic Canada from 2008-2009 to 2013-2014. Based on analysis of administrative data, approximately half of the projects supported over the period were related to expansion/modernization projects to businesses for equipment purchases and facility improvements/expansions. The second largest project type, business support assistance (22%), represents non-repayable assistance to not-for-profits such as industry sector associations and business service centres that in turn, provided services to businesses. The majority of business support projects (90%) focus on promoting entrepreneurship and/or developing the skills and knowledge required to start or grow a business. For a more detailed breakdown of business support activities, see Appendix H.

PBSI funding (14%) consisted of non-repayable contributions to businesses to assist them in hiring needed skilled expertise or meeting training or certification needs. Relatively smaller amounts of assistance in the form of contributions were approved for establishing new businesses (7%) and to undertake market development, marketing plans or activities (6%). However, a data scan revealed that approximately 50 additional projects under the Expansion/Modernization category and 181 PBSI projects also included some marketing-related activities.

Table 3: Approved P&G Projects and Funding, 2008-2009 to 2013-2014

Project Type	Description	Projects (#)	Projects (%)	ACOA Assistance ($M)	ACOA Assistance (%)
Expansion/ Modernization	Expand, modernize, upgrade, renovate facilities, purchase new equipment and machinery, expand sales, acquire technology, working capital	767	28	170.6	51
Business Support	Training, counselling, mentoring to SMEs within certain industry sectors, general SMEs, and special target groups, youth entrepreneurship promotion and supports	615	22	72.2	22
PBSI	Hire expertise, obtain certifications, implement business improvements	1,163	43	47.8	14
Establishment	Establish new facilities, purchase new equipment and technology	97	4	21.5	7
Marketing	Promotional activities, hire marketing expertise	85	3	19.5	6
Total		2,727	100.0	331.6	100.0

Source: QAccess database, May 2014

The amount of assistance approved for these types of projects each year is depicted in Figure 1. The substantial increase in expansion/modernization projects is consistent with management’s decision to increase programming focus on business growth (through the provision of repayable contributions to growing businesses) over the timeframe. Part of the observed increase in expansion/modernization projects during 2012-2013 and 2013-2014 is due to the decision to include all capital cost projects and domestic marketing activities under the new P&G sub-program which has previously been coded to other sub-programs under ED.

Figure 1: ACOA Assistance Approved for P&G Project Types by Fiscal Year

(D)

Source: QAccess database, May 2014

Table 4 depicts the types of clients supported through P&G over the evaluation period. SMEs received 75% of the overall funding (approximately $250 million), in the form of repayable contributions (i.e. debt financing), and non-repayable contributions toward productivity and business skills development. The majority of the remaining approved funds supported universities and colleges (5%; $18 million) or other types of non-profits (17%; $59 million), in their efforts to promote entrepreneurship and provide supports to businesses.

Table 4: P&G Clients and Funding, 2009-2014

Types of Clients	Projects (#)	Projects (%)	ACOA Assistance ($M)	ACOA Assistance (%)
SMEs	2077	76	248.9	75
Other Non-Profit Organizations	263	10	37.9	11
Universities / Colleges	131	5	17.9	5
Community Economic Development and Municipal Organizations	109	4	7.8	2
Industry Associations	63	2	7.2	2
Business Associations / Boards of Trade	42	2	6.1	2
Cooperatives	18	1	3.5	1
Indian Band Council/Native Communities	12	0	1.4	0
Province/Provincial Crown Corporations	8	0	.7	0
Federal Gov’t/Federal Crown Corporation	4	0	.2	0
Grand Total	2727	100	331.6	100

Source: QAccess database, May 2014.

3.0 Findings: Relevance

The relevance of the P&G sub-program was assessed by examining the continued need for the programming and the alignment between the programming, Government of Canada and Agency priorities, and federal roles and responsibilities.

3.1 Continued Need for Programming

Judgment Criteria	Key Finding
Programming-related needs are still present at least to the same degree as they were five years ago, including those related to promoting entrepreneurship, developing business skills, expanding and modernizing a business, and accessing expertise.	While the ability of SMEs to access capital has improved in some ways since the previous evaluation, access to capital continues to be a greater challenge in Atlantic Canada than the rest of Canada. Overall labour productivity, including in manufacturing and other sectors supported by ACOA is lower in many parts of Atlantic Canada than in the rest of the country. Needs exist for entrepreneurship programming and business skills development supports, ranging from general education and awareness about entrepreneurship to more specialized business skills. Key informants indicated that a lack of business skills impedes the success of many Atlantic SMEs. The need for assistance with business succession planning is becoming greater to support continued existence of viable Atlantic businesses.
Evidence of appropriate programming responses to changing landscape and needs, including changes in other programming.	P&G programming is targeted at addressing many of the changing needs in Atlantic Canada, including needs for access to capital and productivity improvements. While annual planning exercises are undertaken, internal stakeholders raised the need for an articulated and communicated P&G strategy, which reflects key objectives, activities and priorities, including entrepreneurship promotion/supports and business skills development activities, as well as commercial programming.
Information on the changing SME landscape in Atlantic Canada is considered in decision making by key stakeholders (i.e., ACOA management).	Information on the changing economic context and changing needs of businesses is considered in decision making. However, decision making could be better supported by greater consultation with external stakeholders, greater linkages with policy divisions and other sources of research, including policy research in certain areas, and strategic intelligence for working with clients.

Judgment Criteria

Key Finding

Programming-related needs are still present at least to the same degree as they were five years ago, including those related to promoting entrepreneurship, developing business skills, expanding and modernizing a business, and accessing expertise.

While the ability of SMEs to access capital has improved in some ways since the previous evaluation, access to capital continues to be a greater challenge in Atlantic Canada than the rest of Canada. Overall labour productivity, including in manufacturing and other sectors supported by ACOA is lower in many parts of Atlantic Canada than in the rest of the country.

Needs exist for entrepreneurship programming and business skills development supports, ranging from general education and awareness about entrepreneurship to more specialized business skills. Key informants indicated that a lack of business skills impedes the success of many Atlantic SMEs.

The need for assistance with business succession planning is becoming greater to support continued existence of viable Atlantic businesses.

Evidence of appropriate programming responses to changing landscape and needs, including changes in other programming.

P&G programming is targeted at addressing many of the changing needs in Atlantic Canada, including needs for access to capital and productivity improvements.

While annual planning exercises are undertaken, internal stakeholders raised the need for an articulated and communicated P&G strategy, which reflects key objectives, activities and priorities, including entrepreneurship promotion/supports and business skills development activities, as well as commercial programming.

Information on the changing SME landscape in Atlantic Canada is considered in decision making by key stakeholders (i.e., ACOA management).

Information on the changing economic context and changing needs of businesses is considered in decision making. However, decision making could be better supported by greater consultation with external stakeholders, greater linkages with policy divisions and other sources of research, including policy research in certain areas, and strategic intelligence for working with clients.

Economic Context

Economic conditions played an important role in influencing the challenges faced by ACOA programming from 2008-2009 to 2013-2014. The global recession starting in 2008 created challenges for all four Atlantic provinces. As of the end of 2014, New Brunswick’s unemployment rate had not yet recovered to pre-recessionary levels, and the gross domestic product (GDP) growth rate was negative or almost negligible over most of the period. In Nova Scotia, by the end of 2014, unemployment rates were higher than they had been in 2008, while GDP grew very little. Prince Edward Island has done slightly better in terms of recovering from the recession, with an unemployment rate that has decreased to be close to that of 2008, and a low but steady GDP growth rate each year.

However, Newfoundland and Labrador’s experience over the period was different from the others, as the province went through a period of unprecedented growth and record low unemployment driven by a number of major projects in the areas of oil and gas, mining, and hydroelectricity. Despite this, Newfoundland and Labrador’s unemployment rate has remained the highest in the country, due primarily to elevated levels of unemployment in many rural areas which are dependent upon declining traditional industries.

Demographic Challenges and Labour Shortages

The Atlantic Canadian economy is shaped by the fact that it has both the oldest and the most rural populations in Canada. In the 2011 Census, among all provinces, the median age of the population was highest in Newfoundland and Labrador, followed by the three Maritime Provinces[xix]. The aging of the population has been exacerbated by outmigration, with this trend highest for those aged 18-24, but also for children and adults up to age 44.

The Atlantic Provinces also have the highest proportion of people located in rural areas, which makes economic development a greater challenge. Compared to the Canadian average where 19% of the population is located in rural areas, the Atlantic Provinces’ rural populations vary from 41% in Newfoundland and Labrador to 53% in Prince Edward Island[xx]. Similarly, a significant proportion of small and medium-sized enterprises are located in rural areas, with a rate of 51% in Prince Edward Island, 34% in Nova Scotia, and 42% in Newfoundland and Labrador, although only 1% in New Brunswick.[xxi]

Consistent with the aging of the population, a shrinking in the number of people in the labour force, including those who are self-employed and other types of labour (skilled and unskilled), is projected for Atlantic Canada in future years and is expected to limit GDP growth in the region in the future. Most projections expect there to be, at minimum, a substantial shrinking in the size of the labour force as people retire. Many also predict shortages of labour, where there will not be enough workers to fill available jobs.[xxii] Currently, the economies of the Atlantic Provinces are experiencing a certain amount of “skills mismatches”, whereby the skills of unemployed workers do not match the skills needed for available jobs. Other factors contribute to this disconnect, such as workers and jobs being available in different geographic areas and the reluctance or inability of employers to increase wages to attract workers. Key informants indicated that finding people with the skills and expertise required was a challenge for SMEs that are ACOA clients. Similarly, in the client survey, 33% of respondents cited “access to skilled workers” as a major challenge to further developing their businesses.

Need for Access to Capital

In the last evaluation of these activities in 2010, it was found that Atlantic Canadian SMEs had a more difficult time than firms in other parts of Canada in obtaining financing for their businesses. Since that time, there is less of a discrepancy, but access to capital remains a challenge for Atlantic businesses nonetheless.

According to Industry Canada’s Survey on Financing and Growth of SMEs,[xxiii] 57.7% of debt or equity financing to Atlantic Canadian SMEs is provided by banks, which is comparable to the Canadian rate of 55.7%. However, governments provide a higher percentage (12.9%) of the financing in Atlantic Canada, compared to the Canadian average (6.5%). The difference seems to be the result of lower rates of credit union financing in Atlantic Canada, which finances 9.7% of Atlantic SMEs.

In terms of debt financing, 24.2 % of SMEs in Atlantic Canada requested this in 2011[xxiv], which compares closely to the national average of 25.5%. However, the average amount requested for financing in Atlantic Canada was lower than in any other province, requests were more likely to be approved partially instead of fully, and the average interest rate charged for debt financing was higher in the Atlantic region (7.1% vs. 6.7% in Canada overall). As such, credit conditions may not be as favourable as in other parts of Canada.

In terms of equity financing, request rates were lower in Atlantic Canada, at only 1.7% of firms, compared to 2.3% for Canada as a whole. The rate of approval of the full amount of equity financing requests was at 53.9% in Atlantic Canada, significantly lower than the national average of 69.3%. “Angel investment”, where an individual or group provides private financing (usually in the form of equity, but it can also be debt) for businesses, is also more limited in Atlantic Canada. A 2013 survey of Newfoundland and Labrador entrepreneurs indicated 2% participation in angel investing versus 3.9% for the rest of Canada.[xxv]

Overall, access to financing remains a challenge for businesses in Canada, and even more so in Atlantic Canada. In the survey of ACOA’s Productivity and Growth clients, access to capital was the most frequently cited major challenge to further developing their businesses, identified by 34% of those surveyed.

Need for Increased Productivity

There continues to be a need for increasing the productivity of Atlantic businesses. In terms of labour productivity (i.e. the value of GDP per hour worked), in 2013, Newfoundland and Labrador ranked second among the provinces in terms of total labour productivity of business sector industries due to thriving oil and gas and mining sectors, while the three Maritime Provinces had the lowest in the country, with New Brunswick ranked 8^th, Nova Scotia 9^th, and Prince Edward Island 10^th.[xxvi] The performance of the Newfoundland and Labrador economy with respect to labour productivity was almost fully due to the oil and gas industry. In many other sectors, including durable and non-durable manufacturing, the province’s productivity rates remain low.

The development of new markets for Atlantic businesses, both inside and outside of Canada, can play a role in increasing productivity by enabling greater economies of scale and increases in sales. Increased sales per worker, in turn, improves SME productivity.

Need for Entrepreneurship and Business Skills Development

According to surveys of the business climate in Canada, the environment for entrepreneurship is viewed as fairly positive, with high levels of people believing that entrepreneurship is a good career choice and that opportunities exist for success.[xxvii] While attitudes towards entrepreneurship may be relatively positive, there is also a lack of knowledge and skills regarding how to start and develop a business. Also, since 2008-2009, business ownership as a proportion of the labour force is lower in Atlantic Canada - from 12.1% to 12.7% - than it is for Canada as a whole - from 15.6% to 16.1%.[xxviii]

In terms of the need for business skills development, the more general needs include awareness-raising and education about entrepreneurship to equip people to be able to identify and successfully develop entrepreneurial opportunities, whether through the formal educational system, the vocational system, and/or training courses. More specific needs include specific business, management and technical competencies, which can often be addressed by means of coaching, mentoring, and business consulting and advisory services.[xxix] Key informants highlighted the need for greater business skills, including cash flow management and general business management, and knowledge that pertains to growing a business (e.g., sales and market development, logistics and distribution).

Extent to which ACOA Addresses Needs

The findings indicate that P&G programming has been aimed at addressing needs. In support of the need for Access to Capital, ACOA’s P&G programming has provided unsecured, interest-free, repayable contributions to Atlantic Canadian businesses in order to help them address difficulties in accessing the capital that would help grow and develop their businesses. P&G programming has also provided funding for organizations that assist businesses in accessing capital and promoting investor readiness, such as support for “angel” investment networks. ACOA’s support has remained focused on both rural and urban clients. The P&G sub-program supported 1767 rural and 960 urban projects within the scope of the evaluation. In total, assistance consisted of $235 million (71%) directed to rural initiatives while $96 million (29%) was directed at urban initiatives.[xxx]

P&G programming has also addressed the need for productivity improvements by enabling the purchase of new, innovative technologies and facilitating access to the specialized expertise that businesses need in order to increase their productivity. For example, P&G has enabled SMEs to gain access to expertise in ‘lean’ manufacturing methods, both through direct PBSI supports to business and by supporting non-profit organizations such as the Canadian Manufacturers and Exporters, which provide the advice to SMEs. ACOA has also supported sales and market development projects to help SMEs, which can support productivity (sales per worker).

ACOA has supported business skills development needs through P&G programming. With respect to direct services to SMEs, PBSI has been a key tool that has been used for the provision of consulting and advisory services for specialized skills (1,163 projects for $21.6 million in approved assistance). ACOA has also provided supports to organizations that offer business skills development initiatives to SMEs both across sectors and within specific sectors and target groups (over 295 projects approved for $42 million over the period). Business skills development activities are also supported through other ACOA sub-programs. For example, similar activities with a focus on rural areas are supported within ACOA’s Community Investment sub-program[xxxi], and initiatives supported under the Innovation and Commercialization sub-program provide advice and skills development for technology-oriented firms.

In addition to the above-described business skills development activities, ACOA supported entrepreneurship initiatives aimed at youth through the P&G sub-program. In particular, the Agency approved 268 projects for over $22 million for initiatives promoting entrepreneurship as a career choice or supporting the start-up of new businesses through training, advice and other types of support (see next section for further detail). Key informants cited entrepreneurship initiatives supported with post-secondary institutions as particularly promising, as they integrate entrepreneurship training with other disciplines and can provide practical experience in starting a business.

By their nature, direct services to businesses under P&G tend to support older firms (client survey results indicate approximately 85% of firms are over five years old) rather than new entrepreneurs, with a substantial focus on capital costs for manufacturing and other sectors. Interviewees explained that new or younger businesses are less likely to meet the financial and experience-related financing criteria, and believed those needs are largely being met by other organizations such as the Community Business Development Corporations (CBDCs) or other not-for-profit organizations that offer micro-credit or other assistance, many of which are supported through other areas of ACOA programming. The Agency’s IC programming has also grown its focus on providing debt financing to start-up businesses, particularly in technology sectors, and also supports non-profit organizations that assist start-up ventures in bringing new technologies to market.

Internal interviewees pointed to the need to map out the landscape of entrepreneurship supports in the region, within and outside ACOA, to determine where important gaps might exist and identify best practices that should be adopted to maximize project impacts. Some ACOA regions have begun this work, and ACOA P&G management are planning an Agency-wide entrepreneurship and business skills review.

To remain informed of evolving needs, ACOA has undertaken research and policy development on key issues, and collaboration or consultations with other departments. However, certain information gaps may require new research, and some key informants identified challenges in accessing policy research and information related to some areas, such as information on key sectors, best practices that improve productivity, investor readiness, sales/market development, and entrepreneurship. They expressed a desire for greater linkages with policy divisions and other sources of information, for greater access to policy research for modifying programs or setting strategy, and strategic intelligence for working with clients.

While both internal and external key informants indicated that ACOA often has very good collaboration and consultative practices with its partners on projects and in certain areas, others identified that it would be beneficial if ACOA adopted an approach of greater consultation with external stakeholders to support decision-making and development of broader Agency policies and directions.

Many internal interviewees indicated there was an absence of an articulated and communicated P&G strategy to guide activities and decision-making. Such a strategy would be helpful in identifying the key objectives, activities and priorities for P&G, and ensuring that activities are targeted toward the achievement of these objectives across the regions, while still preserving a decentralized approach which allows regions to respond to their context and needs. Many interviewees cited the need for identifying an approach for entrepreneurship and business skills development activities and increasing focus in this area, given the growing needs within the Atlantic region.

Programming for Women and Youth Entrepreneurs

Judgment Criteria	Key Finding
Evidence of the nature and extent of ACOA support to youth and women and its evolution over the evaluation period	There was a decline in ACOA support for youth entrepreneurship projects under P&G programming over the period of the evaluation. Funding for initiatives that specifically target women has remained relatively steady over the time period. Internal informants expressed a desire to increase planning and coordination activities in this area.
Evidence of the need for targeted programming strategies for women and youth	Key informants, youth unemployment rates and demographics (higher age of entrepreneurs in Atlantic Canada) support the need for the development of youth entrepreneurship but also business succession planning in order to encourage the continued survival and growth of viable Atlantic businesses following the retirement of business owners. Women continue to represent a potential area of growth for entrepreneurship in Atlantic Canada, and the need for support for women entrepreneurs still exists. There is an opportunity to review the representation of women in the Agency’s commercial programming. Currently, the Agency does not collect gender-disaggregated data to enable such an analysis. The Agency has recently taken steps to explore the further use of Gender-Based Analysis Plus (GBA+) within the Agency, but has not decided how GBA+ will be further incorporated into policy or planning activities.

Judgment Criteria

Key Finding

Evidence of the nature and extent of ACOA support to youth and women and its evolution over the evaluation period

There was a decline in ACOA support for youth entrepreneurship projects under P&G programming over the period of the evaluation.

Funding for initiatives that specifically target women has remained relatively steady over the time period.

Internal informants expressed a desire to increase planning and coordination activities in this area.

Evidence of the need for targeted programming strategies for women and youth

Key informants, youth unemployment rates and demographics (higher age of entrepreneurs in Atlantic Canada) support the need for the development of youth entrepreneurship but also business succession planning in order to encourage the continued survival and growth of viable Atlantic businesses following the retirement of business owners.

Women continue to represent a potential area of growth for entrepreneurship in Atlantic Canada, and the need for support for women entrepreneurs still exists.

There is an opportunity to review the representation of women in the Agency’s commercial programming. Currently, the Agency does not collect gender-disaggregated data to enable such an analysis.

The Agency has recently taken steps to explore the further use of Gender-Based Analysis Plus (GBA+) within the Agency, but has not decided how GBA+ will be further incorporated into policy or planning activities.

The evaluation team was asked to assess the extent to which changes in funding programs directed at youth and women entrepreneurs during the period affected the nature of the projects that were being approved, and whether there was a continued need to provide entrepreneurship or business skills development supports to these groups. Further details are provided in Appendix B.

Support for Youth Entrepreneurs

The Young Entrepreneur Development Initiative (YEDI), which existed between 2001 and 2010, was designed to provide funding for the support of organizations that, in turn, help support the development of entrepreneurship among youth. YEDI’s funding allocations up to 2004-2005 were as high as $5.2 million annually, and decreased to $2.5 million annually after that. After 2009-2010, senior management decided that YEDI would no longer exist as a separate funding envelope, but its projects would be eligible for funding through the business support element of ACOA’s Business Development Program.

An analysis of all non-commercial project descriptions within P&G programming revealed that over the course of the six years of the evaluation period, total projects aimed at promoting youth entrepreneurship or providing supports to youth for starting a business represented 31% ($22.2 million) of all P&G Entrepreneurship and Business Skills Development (EBSD) funding.[xxxii] Throughout the period, there was a decline in funding approved for youth entrepreneurship and support activities, from a high of $4.8 million in 2009, to $1.7 million in 2014. The number of projects approved each year decreased from 57 to 34.[xxxiii] The evaluation did note that young entrepreneurs may be accessing supports under P&G projects that are targeting the general SME community; and as stated in the previous section, the Agency does support projects outside P&G programming that target new businesses under both Innovation and Commercialization (e.g., incubators and accelerators) as well as Community-Based Business Development sub-programs (e.g., debt financing for business start-ups and skills development under the Community Futures of Tomorrow program).

A number of key informants indicated that there has also been a decrease in coordination and discussion between regions regarding programming for women and youth. They noted that working groups for YEDI and WBI had been useful in encouraging cross-regional discussions and information exchange, but were discontinued with the end of the funding envelopes. As confirmed by management, a strategic decision to decrease focus on these targeted funding envelopes naturally led to less emphasis and resources being allocated in this area.

As stated in the previous section, research indicates that the need for support and development of young entrepreneurs has increased. Not only do the Atlantic Provinces have the oldest populations in the country, and continued out-migration of youth, but the four provinces also have the highest unemployment rates for youth amongst any of the provinces across Canada: approximately 16% in Prince Edward Island and Newfoundland and Labrador, and 18% in New Brunswick and Nova Scotia.[xxxiv] Atlantic Canada also has the largest proportion of SMEs owned or run by people over age 50. In fact, the Atlantic Provinces have more SMEs headed by people over the age of 65 than under the age of 40.[xxxv] Key informants and demographics point to a need for the development of youth entrepreneurship as well as business succession planning in order to encourage the continued survival and growth of viable Atlantic businesses following the retirement of business owners.

Support for Women Entrepreneurs

Like YEDI, the Women in Business Initiative (WBI) funding envelope was designed to improve the growth and competitiveness of women-owned businesses from approximately 2000-2001 to 2009-2010. After that year, it was decided that the WBI would no longer exist as a separate funding envelope, but that its projects would be eligible for funding through ACOA’s other mechanisms such as the BDP. The Agency also supports women-owned businesses through its Community-Based Business Development sub-programs (e.g., debt financing for business start-ups and skills development under the Community Futures of Tomorrow program).

A review of all non-commercial project descriptions over the course of the six-year evaluation period revealed that non-commercial projects targeting women business owners represented 19% ($13.6 million) of all P&G EBSD funding. Overall Agency funding levels for these projects have remained fairly consistent over the time period examined, with approximately $2 million in approvals per year. Prior to the evaluation period, the annual allocations to WBI were much higher ($6.35 million per year for three years starting in 2002-2003, and $2.5 million per year subsequently), but in practice, annual approvals had been only slightly higher than during the evaluation period. The number of projects approved annually declined consistently from 17 approved in 2008-2009, to eight approved in 2013-2014.[xxxvi] Each region saw a decline of 1 or 2 projects a year from the beginning of the period to the end. Although funding levels have remained steady, key informants expressed a desire to increase planning and coordination activities in this area in order to mitigate risks to ongoing focus and support.

The need for support of women entrepreneurs still exists, as the economic conditions for women in Atlantic Canada are similar to those identified when WBI was initiated in 2000. Women are still under-represented as entrepreneurs, and are the largest “under-represented” group[xxxvii] (compared to their proportion of the population) in the Atlantic labour force, including the self-employed. In 2010, women owned 18% of businesses with under 100 employees in Atlantic Canada, and co-owned an additional 8%. [xxxviii] In Canada, women’s businesses tend to be smaller, both in terms of revenue and in terms of employees.[xxxix] However, women’s businesses likely have higher survival rates, and they stay in business longer.[xl] The number of self-employed women in Atlantic Canada steadily increased over the evaluation period, while the number of self-employed males stayed fairly steady. Increasing the participation of under-represented groups and reducing barriers to their participation in the economy and in the labour force have often been cited as important components of a strategy to address the potential looming shrinkage of the labour force and skills shortages in Atlantic Canada.[xli]

Internal key informants noted that the continued support of non-profit organizations that assist women entrepreneurs is important. There is also an opportunity to review the representation of women in the Agency’s commercial programming. Women entrepreneurs in Atlantic Canada are predominantly in services-producing sectors, while the majority of ACOA’s direct supports to SMEs are to good-producing sectors, such as manufacturing. ACOA has established sector eligibility criteria with the goal of maximizing economic impacts to the region, corresponding with the Agency’s mandate. Nonetheless, women comprise only 11% of self-employed persons in the goods-producing sector in Atlantic Canada[xlii]. Currently, the Agency does not collect gender-disaggregated data to enable such an analysis.

Gender-Based Analysis Plus (GBA+) is a method of analysis that assesses the impact of initiatives on diverse groups of women and men, and the “plus” means that the analysis goes further to consider other identity factors such as age, language or ethnicity.[xliii] The Government of Canada expects GBA+ to inform the development of policies, programs, and legislation.[xliv] ACOA has recently taken steps to explore the further use of GBA+ within the Agency, by arranging training sessions for some of its staff. It has not been decided how GBA+ will be further incorporated into policy or planning activities.

3.2 Alignment with Government Priorities, Federal Roles and Responsibilities

Judgment Criteria	Key Finding
There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome, priorities and strategy.	P&G activities are well-aligned with ACOA’s strategic outcome of a competitive Atlantic Canadian economy. P&G programming is aligned with federal government priorities, including creating jobs and opportunities for Canadians, as well as promoting small business and tourism.
P&G programming is aligned with ACOA’s roles and responsibilities as outlined in the ACOA Act.	P&G programming is well-aligned with the ACOA Act.

Judgment Criteria

Key Finding

There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome, priorities and strategy.

P&G activities are well-aligned with ACOA’s strategic outcome of a competitive Atlantic Canadian economy.

P&G programming is aligned with federal government priorities, including creating jobs and opportunities for Canadians, as well as promoting small business and tourism.

P&G programming is aligned with ACOA’s roles and responsibilities as outlined in the ACOA Act.

P&G programming is well-aligned with the ACOA Act.

There is a strong alignment between ACOA’s P&G sub-program and the Agency’s strategic outcome of a competitive Atlantic Canadian economy. P&G activities reflect the organization’s priorities, which include assisting businesses to become more innovative, productive and competitive, as well as implementing strategies in response to the distinct economic needs of Atlantic Canada, with a particular focus on rural business development[xlv].

Programming aligns with the high-level outcome areas for government spending, as defined by the whole-of-government framework, including “strong economic growth”, “income security and employment” and “an innovative and knowledge-based economy”.[xlvi]

With respect to federal priorities, P&G programming is aligned with the themes of the 2013 Speech from the Throne,[xlvii] including creating jobs and opportunities for Canadians, advancing international and inter-provincial trade, and promoting small business and tourism. Similarly, the 2011 Speech from the Throne[xlviii] emphasized the creation of conditions to support growth and job creation, including support for innovation and new technologies.

P&G programming is well-aligned with the ACOA Act, as it supports the achievement of the Agency’s strategic objective, which is “to support and promote economic development of Atlantic Canada, with particular emphasis on small and medium-sized enterprises….”[xlix] It is also aligned with the types of activities mandated under the act, which includes support for enterprises, especially SMEs, the development of entrepreneurial talent, and support for business associations and other activities related to improving the business environment. Several other regional development agencies support similar types of programming.[l]

Extent of Duplication, Overlap and Complementarity

Judgment Criteria	Key Finding
ACOA P&G programming complements (rather than duplicates or overlaps with) other programming, including other federal and provincial governments.	P&G programming is complementary to funding provided by other funders. P&G programming is part of a complex landscape of programs offered in the region, which can be a challenge for both funders and entrepreneurs in terms of awareness and understanding.

ACOA is one of many organizations that comprise the complex landscape of programming and supports targeting entrepreneurs in Atlantic Canada. While funding is available through a number of sources, the majority of the programming is very distinct in terms of who they target and where they fall within the financing continuum (Appendix I).

The findings indicate the complexity of the funding landscape means that it can be difficult for both entrepreneurs and funding providers to be aware of and understand the nature of the many different programs, and how to best access funding. Many external interviewees viewed ACOA as having a leadership role among the different funders, while many indicated that they did not fully understand the types of programming offered by ACOA. They also indicated that prospective clients (SMEs and not-for-profits) are having similar difficulties knowing about and understanding the wide range of programming that exists. A few interviewees stated that information services provided through the Canada Business Network are helpful in that regard.

Taking into consideration the complex landscape of funding available, there does not appear to be any visible areas of overlap. Consistent with what was reported in the 2010 Financing Continuum (FC) Evaluation, ACOA’s programming is complementary to other programming, largely due to ACOA’s unique funding characteristics related to unsecured, interest-free financing.

4.0 Findings: Performance – Effectiveness

The effectiveness of the P&G sub-program was assessed by examining: (1) incrementality; (2) evidence of achievement of expected outcomes; (3) integration of ACOA ED sub-programs and their effect on outcomes; and, (4) barriers, lessons learned, facilitators or best practices for achieving outcomes.

4.1 Incrementality

Judgment Criteria	Key Findings
The impact of the absence of the programming is similar to or greater than that observed in the previous evaluation.	ACOA’s P&G programming has had a positive incremental effect on the achievement of expected outcomes. Without ACOA financial assistance, it is likely that many projects would not have moved forward, and there would have been negative consequences on the scope, quality and timing of those projects that would have proceeded. ACOA provides a substantial proportion of total project costs for PBSI projects (62%) and non-commercial projects (42%). While the proportion of costs supported by ACOA is smaller for repayable contributions, it has increased since the previous evaluation. This, combined with decreases in leveraging for non-commercial projects, supports the continued importance of ACOA’s P&G programming.

Judgment Criteria

Key Findings

The impact of the absence of the programming is similar to or greater than that observed in the previous evaluation.

ACOA’s P&G programming has had a positive incremental effect on the achievement of expected outcomes. Without ACOA financial assistance, it is likely that many projects would not have moved forward, and there would have been negative consequences on the scope, quality and timing of those projects that would have proceeded.

ACOA provides a substantial proportion of total project costs for PBSI projects (62%) and non-commercial projects (42%). While the proportion of costs supported by ACOA is smaller for repayable contributions, it has increased since the previous evaluation. This, combined with decreases in leveraging for non-commercial projects, supports the continued importance of ACOA’s P&G programming.

Incrementality is a measure of a project proponent’s ability to proceed with a project with the same scope, within the expected time frame, without ACOA assistance. The measure of incrementality demonstrates that the outcomes being reported are attributable, in part, to the Agency’s P&G activities and are less likely to have been achieved otherwise.

One of the factors that impacts the incrementality of ACOA support is the level of assistance a client receives from ACOA rather than other sources.[li] According to program guidelines, the maximum rate of ACOA contribution is 50% for repayable loans, with the exception of marketing projects, which can receive 75% of total project costs. The maximum rate of contribution is 75% for PBSI projects.

ACOA contributed 35% of total project costs for commercial projects over the period. ACOA’s proportion of funding was 32% for repayable contributions, up from 27 per cent since the previous evaluation, suggesting that the importance of ACOA’s role has slightly increased for repayable contributions. The proportion of ACOA funding for PBSI projects was 62% for PBSI, similar to the previous evaluation (61%).

For non-commercial projects, ACOA contributed 42% of total project costs (FC 55%; EBSD 46%), representing a decrease of 7 percentage points since the previous evaluation. Provincial governments, which provided 12% of project costs, are sometimes the only other external funding source for these projects, while the remainder is usually contributed from the organization’s revenues or cost-recovery activities. While the proportion of funding coming from other sources has increased since the previous evaluation, the contributions from provincial governments declined between fiscal years 2008 and 2014. Key informants given the Provinces’ increasing fiscal challenges,the Agency will likely experience pressure to provide greater amounts of funding for similar projects in the coming years.

According to the survey results (Table 5), ACOA support had a greater impact on non-commercial clients than on commercial clients with respect to the proportion of clients who would have cancelled their projects in the absence of the Agency’s support. Without ACOA funding, the scope of the projects for repayable contributions and non-commercial projects would have been affected more than for PBSI projects, and only a small proportion of projects would have proceeded as planned without ACOA assistance.[lii]

Table 5: Incrementality as Rated by Clients Surveyed

Outcome that would have occurred in the absence of ACOA funding	Commercial		Non-Commercial
	Repayable Contributions	PBSI	Non-Commercial
Total Answering	n=140	n=100	n=38
Project would have been cancelled	20%	30%	45%
Project scope would have been reduced	41%	33%	39%
Project start would have been delayed	27%	30%	3%
Project duration would have been longer	31%	17%	8%
Project would have proceeded as planned	9%	5%	8%

Source: Client Survey, May 2014

External and internal key informants indicated that ACOA funding influences other partners to support P&G commercial projects, mainly because ACOA does not take security on contributions, thereby reducing the risk undertaken by commercial lenders (50 per cent of surveyed commercial clients also received funding from commercial lenders). In the case of non-commercial projects, some other funding agencies were said to often rely on ACOA’s evaluation and decision for determining their own participation.

Similar to the findings of the last evaluation, leveraging per ACOA dollar invested is strongest for commercial, repayable-contribution projects ($2.13), down from the previous evaluation ($3). Leverage per dollar invested is $0.60 for PBSI projects, slightly less than the previous evaluations ($0.63). Key informants reported that in most cases the firm will support the remainder of the costs as other lenders do not typically support the types of projects undertaken by PBSI. Non-commercial projects leveraged $1.38 from other sources, up from the previous evaluation ($1.18 EBSD; $0.80 FC).

4.2 Achievement of Expected Outcomes

Based on an analysis of all lines of evidence, P&G programming is contributing to the achievement of expected outcomes for SME clients and beneficiaries of ACOA-supported, non-commercial projects.

Judgment Criteria	Key Findings
Performance targets have been established and achieved.	For commercial programming, the achievement of outcomes was supported by establishing and meeting a priority to increase direct supports to business. This resulted in an increase from $15.4 million to $34.5 million in approved repayable contributions over the first four years of the time frame.
There is sufficient evidence to argue the contribution of the sub-program activities to the achievement of outcomes .	Most important areas of outcome achievement for both repayable contributions and PBSI projects include increased production capacity, improvements in product quality and productivity, sales, competitive position and incremental post-project employment. Notable PBSI results also include the development of management and marketing skills. ACOA-supported SMEs out-performed unassisted SMEs in sales growth, labour productivity growth, and in survival rates. Amount of contributions repaid as a percentage of total receivables was 84%. For non-commercial programming, an annual average of 87% of beneficiaries reported increasing their business knowledge and skills, exceeding performance targets of 75%. Organizations that supported businesses reported increasing awareness of entrepreneurship, SME marketing capacity and activities, business knowledge and skills of aspiring and existing SMEs, networks and partnerships, and SME growth and competitive position. Key informants suggested ACOA work to further develop outcomes related to entrepreneurship, productivity, and sales and market development.

Judgment Criteria

Key Findings

Performance targets have been established and achieved.

For commercial programming, the achievement of outcomes was supported by establishing and meeting a priority to increase direct supports to business. This resulted in an increase from $15.4 million to $34.5 million in approved repayable contributions over the first four years of the time frame.

There is sufficient evidence to argue the contribution of the sub-program activities to the achievement of outcomes .

Most important areas of outcome achievement for both repayable contributions and PBSI projects include increased production capacity, improvements in product quality and productivity, sales, competitive position and incremental post-project employment. Notable PBSI results also include the development of management and marketing skills. ACOA-supported SMEs out-performed unassisted SMEs in sales growth, labour productivity growth, and in survival rates. Amount of contributions repaid as a percentage of total receivables was 84%.

For non-commercial programming, an annual average of 87% of beneficiaries reported increasing their business knowledge and skills, exceeding performance targets of 75%. Organizations that supported businesses reported increasing awareness of entrepreneurship, SME marketing capacity and activities, business knowledge and skills of aspiring and existing SMEs, networks and partnerships, and SME growth and competitive position.

Key informants suggested ACOA work to further develop outcomes related to entrepreneurship, productivity, and sales and market development.

4.2.1 Commercial Programming

Data collected from ACOA administrative and financial data systems, survey participants, interviews, and Statistics Canada indicate that commercial P&G programming is achieving its desired outcomes. The greatest impacts for both repayable contribution and PBSI clients appear to be in areas related to production (e.g., volume, quality, range of products, and productivity), as well as sales growth and overall competitive position of the SME client. Notably, PBSI projects also resulted in increases in management and marketing skills. Commercial projects also had an impact on the creation and maintenance of jobs. ACOA-supported SMEs outperformed non-clients in areas of sales growth, productivity growth and business survival. Achievement of SME outcomes related to reaching new markets and entering supplier development chains was not as pronounced. Similarly, program stakeholders interviewed indicated that accessing new markets remains a challenge, especially given transportation costs to and from Atlantic Canada.

Increase in Direct Supports to Growing Businesses

To support the achievement of outcomes for commercial programming, management established a priority to increase direct supports to growing businesses. Accordingly, an analysis of administrative data for P&G projects between 2008-2009 and 2011-2012[liii] reveals that there was a 124% increase in repayable contributions approved over the period, from $15.4 million to $34.5 million. Most of these projects consisted of expansions and modernizations. See Figure 1 for a visual representation of this increase.

Project Activities and Clients

In interpreting the achievement of project outcomes, it is important to understand key characteristics (e.g. firm age, size and sectors) of the project activities and participants.

SMEs that participated in the survey (n=242) were asked to identify areas of focus for their projects, and were able to select more than one response, as most often, projects have several components (Table 6). Common areas of focus for repayable contributions included capital and new technology investments, while PBSI projects had a greater focus on human resource projects (e.g. hiring or training), as well as continuous-improvement types of planning activities such as productivity improvement initiatives and the obtaining of certifications. With respect to marketing and promotions, both types of projects had equal focus on the development of marketing plans, however repayable contributions had a slightly greater focus on the implementation of marketing activities, while PBSI projects had a greater focus on conducting market research and collecting market intelligence.

Table 6: Types of Activities Performed in Repayable Contribution and PBSI Projects

Type of Activity Involved in Commercial Projects	Repayable Contribution	PBSI
Total Answering	n=133	n=99
Capital investment/New technology acquisition	76%	17%
Capital investment in facilities	66%	8%
New technology acquisition	35%	12%
Start-up of a new business operation	29%	19%
Marketing and promotion	38%	42%
Marketing strategy/plan	26%	25%
Market research/business intelligence	17%	29%
Increased investment in marketing activities	20%	16%
Advertising / promotions	17%	12%
Trade shows	12%	7%
Development of new packaging	11%	4%
Product or equipment demonstrations	5%	3%
Human resource investments	25%	43%
Hiring technical or tech management skills	14%	28%
Technical training	17%	17%
Business management training	4%	8%
Planning/Research/Certification	24%	43%
Productivity improvement program	17%	29%
Obtaining a certification (e.g. ISO)	5%	16%
Feasibility / research studies	6%	4%
Environment management program	2%	4%

Source: Client Survey, March 2015

Firm Age

According to surveyed clients who reported their firm age (n=223/242 SMEs surveyed), ACOA SME clients are older firms, with over half of firms (57%) in operation for 15 years or more, 14% of firms in operation between 10 and 14 years, and another 15% of firms in operation between five and 10 years. A relatively small proportion (14%) of surveyed firms reported being in operation under five years.

Firm Size

As indicated by 219 surveyed clients who reported their firm size, ACOA SME clients are relatively small, with two thirds (66%) of client firms having less than 25 employees, another 16% having between 26 and 50 employees, 11% having between 51 and 100 employees, and only 7% of firms with more than 100 employees. These figures would indicate that ACOA client firms are slightly larger than the population of Atlantic SMEs. For example, 93% of firms in Atlantic Canada have less than 20 employees[liv], while firms of this size make up less than 66% of the Agency’s P&G clientele.

Firm Sectors

According to ACOA administrative data, the largest proportion of approved funding (58%) for repayable contributions went to manufacturing sectors[lv], including, for example, manufacturing of food products, metal products and wood products. The next largest sectors were agriculture, forestry, fishing and hunting[lvi] at 15% of funding and professional, scientific and technical services[lvii] at 8%. Other sectors included professional services (8%), accommodations (7%), and supports for mining and oil and gas (6%).

Achievement of Outcomes

Surveyed business clients were asked to rate the extent to which the project supported the achievement of outcomes (Figure 2). Outcomes of repayable contributions that were rated highest included increases in production volume and quality, as well as productivity, overall competitive position, and sales in existing markets. Outcomes of PBSI projects that were rated highest included increases in productivity, production volume and quality followed by improvements in competitive position and management skills.

Figure 2: Average Rating of Outcomes for Repayable Contributions and PBSI projects (1 = “Not at all” to 5 = “To a Great Extent”)

Average Rating of Outcomes for Repayable Contributions and PBSI projects

(D)

Source: Client Survey, March 2015

The achievement of these outcomes are further elaborated upon in the following sections:

Production Capacity, Product Quality, Product Diversity and Productivity

In terms of projects intended to influence production-related outcomes, 76% of repayable contribution clients surveyed and 17% of PBSI clients surveyed undertook projects involving capital investments in production facilities or the acquisition of new technologies. Projects undertaken by 17% of repayable contribution clients and 29% of PBSI clients involved implementing a productivity improvement initiative.

The expansion of production capacity was the outcome highest-rated by clients, with 77% of repayable contribution clients and 68% of PBSI clients reporting that this was a result of their projects (i.e. provided a rating equal to or greater than 3 or “to some extent”). A majority (76%) of repayable contribution clients and 72% of PBSI clients reported that the project resulted in improvements to the quality of products produced. Many (62% repayable contributions, 53% PBSI) also reported that the project increased the range of products they produced.

All lines of evidence support the key role that ACOA has played in increasing the productivity of SME clients. The majority (72%) of repayable contribution clients and 72% of PBSI clients reported that their projects increased the productivity and efficiency of their operations. Clients (n=88) estimated that the average savings per client in the past fiscal year was $150,000, due to productivity improvements associated with the project.

Using data obtained from Statistics Canada, a comparison of labour productivity change over a five-year period (2007 to 2012) was conducted between SMEs that received direct supports under P&G, and all other Atlantic Canadian businesses in the same sectors. Labour productivity in firms supported under ACOA’s P&G sub-activity, which is estimated using nominal sales per worker, rose by 7.1% per year in ACOA-assisted firms, outpacing the increase of 2.0% per year registered by the group of unassisted firms. These results are explained by the fact that sales growth for ACOA assisted firms exceeded employment growth to a greater extent than it did for the comparison group. These results are positive, though some degree of caution must be applied to interpreting the comparison results, given possible differences between ACOA-supported SMEs and the comparison group.[lviii]

Given that productivity measures are known to vary with firm size, the analysis was conducted for small firms (under 20 full-time equivalents, or FTEs), medium-sized firms (between 20 and 100 FTEs), and large firms (over 100 FTEs). The greatest annual productivity increase for ACOA SMEs was for the largest firms (average 11.1% per year), compared to a decline of 7.8% for unassisted firms. Sales per worker for ACOA-supported small firms increased by an average of 6.8% per year, compared to a 0.4% increase for unassisted clients. Results were not as strong for medium-sized ACOA-supported firms, with an average 2.6% increase per year, the same increase observed for unassisted firms. Part of the explanation for the ACOA results by firm size is that sales growth was not as strong for the medium-sized ACOA-supported firms, which impacted the productivity measure for that group. Reasons for the smaller increase in sales were not clear.

Skills, Networks, Partnerships and Strategic Alliances

The projects also contributed to the further development of business management skills, as reported by 43% of repayable contribution clients and 70% of PBSI clients. The improved skills included skills related to general operations, marketing and market intelligence, and business management/development. In addition, repayable contribution clients enhanced client management and development skills, as well as project management skills, while PBSI clients reported enhanced planning skills.

Slightly fewer repayable contribution clients (41%) and a third of PBSI clients reported that technical or technology management skills were enhanced as a result of the project. Clients enhanced skills in the areas of software for operating business management, quality assurance, equipment operation, and general computer skills, while PBSI project clients reported that they improved their skills in online marketing and software development.

The projects also served to expand existing networks, partnerships and strategic alliances (55% of repayable contribution clients and 46% of PBSI clients).

Sales in Existing Markets, Development of New Markets, Marketing and Participation in Supply Chains

All lines of evidence support the key role that ACOA has played in increasing sales by SME clients in existing markets. The majority (74%) of repayable contribution clients reported that their projects resulted in increased sales in existing markets, as did 61% of PBSI clients. Clients who indicated the project helped increase sales were asked to provide an estimate of incremental impacts achieved in their most recent fiscal year, attributable to the project. Those who provided an estimate (n=85) reported a total of $60.2 million in revenues attributable to the project in their most recent fiscal year, for an average of $708,235.29 per client.

According to a comparison of sales increases over a five-year period (2007 to 2012) between ACOA-supported SMEs and non-clients in the same sectors using data obtained from Statistics Canada, ACOA clients’ nominal sales increased on average by 7.8% per year, compared to an average annual increase of 0.2% for unassisted firms. As reflected above, sales growth was greatest for small and large ACOA-supported firms.

The development of new markets for products was also an outcome of many P&G projects, as indicated by 61% of repayable contribution clients and 54% of PBSI clients. Many effectively increased their participation in supply chains (44% repayable contribution clients and 33% PBSI).

While a majority (77%) of 130 responding clients indicated that they had previous exporting experience, 24% of those who had not previously exported products reported that they began exporting as a result of the project.

Many P&G projects involved marketing and promotional activities. These activities were part of 38% of repayable contributions projects and 42% of PBSI projects. This included the development of marketing strategies or plans, the conducting of market research, and investments in marketing activities.

Internal program stakeholders noted that ACOA is assisting clients in marketing and noted special initiatives to increase supply chain participation, but also expressed a desire to further enhance market and sales development supports to SMEs and to further assist them in increasing supply-chain participation.

Competitive Position

P&G clients also reported that their projects improved their firm’s competitive positions; 72% repayable contribution clients and 73% PBSI indicated that the nature of the improvement came about because of increased productivity (18%), improved quality of their products or services (18%) and increased sales (15%). PBSI clients indicated that their competitive position was enhanced especially due to increased productivity (21%) and increased general capacity (16%).

Business Survival

The business survival rate can be used to measure ACOA’s performance in terms of competitiveness. The five-year business survival rate for ACOA-assisted firms under the P&G sub-program is higher than the rate for unassisted firms[lix]. In fact, each year, the business survival rate for ACOA firms supported under P&G exceeded that of the comparison group, and was 92% after the fifth year following start-up for the 2002 to 2012 period, compared with 67% for unassisted firms. This represents a variance in the five-year business survival rate of 25 percentage points between ACOA-assisted firms and unassisted firms.

Repayment of ACOA Contributions

One indicator of whether ACOA-supported firms are able to survive or remain in a relatively good financial position, is their ability to repay ACOA contributions. The proportion of repayable contributions collected against total receivables over the six-year period is 84%. In 2011-2012, twenty-two accounts valued at $2.5 million were written off, which positively impacted the repayment rate for subsequent years.

Creating and Maintaining Employment

P&G projects also resulted in the hiring of staff and often continued employment after the end of the project. Seventy-three per cent of repayable contribution clients and 76% of PBSI clients who responded to this survey question reported that staff was hired or assigned specifically to their projects, with 7.7 and 1.8 average FTEs per project respectively. A large proportion of this staff (74% repayable contribution clients, 52% PBSI) consisted of new hires, and many of these new hires were kept on by the organization after the project was completed (67% repayable contribution clients, 75% PBSI).

Beyond the employees that were directly involved in the project, 163 clients reported that in the most recent fiscal year, a combined total of 4, 210 FTE positions would not have been employed if the project had not been undertaken. (Note: one outlier reported 3,000 FTE positions). One hundred and twenty-nine of these SMEs reported salaries for that year representing a total of $152.9 million in salaries in that year ($44.9 million excluding the outlier), representing an average salary of $37,652, or $42,317 (excluding the outlier).

4.2.2 Non-Commercial Programming

Data collected from ACOA administrative and financial data systems, survey participants, and interviews, suggest that non-commercial P&G programming is achieving desired outcomes. Though the evaluation was unable to survey the SMEs that benefited from these activities, participant exit survey data collected on some of these projects, combined with surveys of the business-serving organizations suggest that SMEs who participate in the services increase their skills knowledge and networks as a result.

Project Activities

Not-for-profit clients who participated in the survey (n=38) were asked to identify areas of focus for their projects, and were able to select more than one response, as most often, projects have several components (Table 7). While the number of respondents to this question represents a small group of organizations (n=35), these responses provide a general sense of the activities performed.

Table 7: Types of Activities (Non-Commercial Projects) and Project Target Client Groups

Type of Activity	n=35
Delivering workshops and training	74%
Events and activities that promote awareness of entrepreneurship (e.g. awards, trade shows)	60%
Providing mentorship and/or advisory services	49%
Assisting with business planning	45%
Increasing access to information on government services, programs, regulations and business resources	40%
Conducting planning and feasibility studies	23%
Staging business planning competitions or related activities	17%
Assisting businesses in obtaining certifications (e.g. ISO certification)	6%
Target Client Group	n=35
Existing businesses/managers	57%
Aspiring and potential entrepreneurs	43%
Business start-ups	40%
Students	37%
Others	17%

Source: Client Survey, March 2015

As indicated in Table 7, the main activities undertaken as part of non-commercial projects included delivering workshops and training, hosting events and activities that promote entrepreneurship, and providing mentorship or advisory services. Key target groups included existing businesses, aspiring entrepreneurs and business start-ups or new businesses.

Achievement of Outcomes

Surveyed non-commercial clients (n=38) were asked to rate the extent to which the project supported the achievement of outcomes. The outcomes of non-commercial projects that were rated highest included increasing entrepreneurship awareness, increasing marketing capacity or activities, enhancing networks and partnerships as well as overall competitive position, and increasing business knowledge and skills for aspiring and existing entrepreneurs (Figure 3).

Figure 3: Average Rating of Outcomes for Non-Commercial Projects (1 = “Not at all” to 5 = “To a Great Extent”)

(D)

Source: Client Survey, March, 2015

Entrepreneurship Awareness and Business Skills and Knowledge of Aspiring and Existing Entrepreneurs

A majority (74%) of non-commercial survey respondents reported that their project increased awareness and interest in entrepreneurship among potential entrepreneurs (average rating 4.29/5), and 61% reported that potential entrepreneurs were provided with the skills needed to plan and launch a new business (average rating 3.42/5). This finding is consistent with the proportion of projects that focused on aspiring entrepreneurs (43%) and students (37%), as well as the proportion of projects that involved entrepreneurship promotion activities (60%).

Over half of the non-commercial clients surveyed (59%) reported increasing business skills among existing SMEs, which is a positive result considering the proportion of projects that targeted existing SMEs (57%). In terms of specific business skills, clients reported enhancing SME marketing capabilities and activities (71%), financial planning (50%), market intelligence (28%), business planning (17%), human resources (17%), management (11%), sales (11%), and product design/production (11%).

Nearly half (47%) of non-commercial clients indicated increasing awareness of government services, programs, regulations and business resources.

As part of on-going performance measurement for some of its EBSD projects, ACOA requires not-for-profit organizations it supports to conduct participant exit surveys in an effort to capture the impact of activities from the perspectives of existing and aspiring SMEs that participate in them (i.e. the project beneficiaries).

Beneficiaries of non-commercial projects were asked to indicate whether the activity increased their knowledge and skills as relates to starting or growing a business. On average, 87% of clients indicated an increase, and the Agency target of 75% was exceeded for each year under the scope of the evaluation (Table 8).

Table 8: Impact on Beneficiaries’ Business Skills and Knowledge

	Fiscal Year
EBSD Performance Data	2008-2009	2009-2010	2010-2011	2011-2012	2012-2013	2013-2014
Increase in business knowledge and skills (Target set at 75%)	79%	87%	87%	90%	85%	96%
Number of Respondents	11,982	7,165	964	12,253	9,597	8,603

Source: Departmental Performance Reports, Fiscal Year 2009 to 2014

Key informants reported that this area of programming continues to increase the awareness of entrepreneurship as a career option, and enhance the business skills and knowledge of aspiring and existing entrepreneurs. They cited the importance of continuing to support business services across sectors (even those that are not supported through commercial programming), while noting the benefits of supporting industry sector associations so they may grow their capacity to serve SME clients and advance specific sectors in Atlantic Canada. Some stakeholders suggested ACOA continue to develop its supports for entrepreneurship and business start-up.

Networking and Partnerships

The majority of clients surveyed (71%) reported increasing partnerships and networking among their beneficiaries (average rating: 4.13/5). Networking activities increase the ability of SMEs to create relationships with potential business partners or other resources that can assist them in further growing their business.

Other Beneficiary Outcomes

Non-commercial clients also rated the extent to which their projects helped SMEs achieve a number of longer-term outcomes or outcomes that are further removed from the assistance provided. Many reported that SMEs that participated in their projects further developed new and existing markets (55%: average rating of 3.23/5); improved operational practices and processes (34%: average rating of 2.48); achieved further growth (53%: average rating 3.41); were able to survive a set-back (47%: average rating 3.11); improved their competitive position; and were able to start a business (34%: average rating 2.75); improved their competitive position (83%: average rating 3.5); and, enabled the start-up of their business (34%: average rating 2.75). Six of these non-commercial clients reported helping to launch a combined 113 new businesses.

4.3 Client-Centred Approach and Influence on Outcomes

Judgment Criteria	Key Findings
Evidence that ACOA’s approach to program delivery enables SMEs to access a range of services across ED programming as required. There is evidence to suggest that this approach enhances a client’s ability to achieve outcomes.	Data analysis showed that some P&G clients are also accessing funding under the IC and IBD sub-programs. ACOA management and staff report that a substantial proportion of P&G clients participate in trade missions organized by non-commercial entities under IBD, and there is strong integration of ED, Community Development (CD) and PAC with respect to specific initiatives such as supplier development. In the 2015 IBD evaluation, IBD clients who also received assistance under P&G rated the extent to which the P&G project impacted SME outcomes as high (average rating: 4.08/5).

Judgment Criteria

Key Findings

Evidence that ACOA’s approach to program delivery enables SMEs to access a range of services across ED programming as required.

There is evidence to suggest that this approach enhances a client’s ability to achieve outcomes.

Data analysis showed that some P&G clients are also accessing funding under the IC and IBD sub-programs. ACOA management and staff report that a substantial proportion of P&G clients participate in trade missions organized by non-commercial entities under IBD, and there is strong integration of ED, Community Development (CD) and PAC with respect to specific initiatives such as supplier development.

In the 2015 IBD evaluation, IBD clients who also received assistance under P&G rated the extent to which the P&G project impacted SME outcomes as high (average rating: 4.08/5).

Analysis of ACOA’s administrative data revealed that 25% of P&G clients are accessing funding from other ED sub-programs (Figure 4). The largest visible connection was with P&G and IC, where 243 clients (15%) received support for projects under both. There was less of a connection between P&G and IBD (6.8% or 110 clients), and an even smaller number of clients (3.5% or 56 clients) had projects under all three sub-programs.

Figure 4: P&G Clients Accessing Other ACOA Funding Programs, (2009-2014)

(P&G clients who also access IBD funding is likely understated in the above diagram as, in many cases, the clients under IBD are non-profit organizations that organize the trade missions rather than the SMEs who participate in them. In these cases, participation of Agency P&G commercial clients in IBD trade missions is not systematically tracked throughout the Agency. One region that tracked it indicated that approximately 60% of IBD trade mission participants were P&G clients. Other examples of integrated programming include special issues that receive concerted focus from PAC, CD, and ED sub-programs, such as economic development activities to mitigate impacts of the Ferry Closure in Yarmouth, N.S., and the Newfoundland and Labrador initiative to support SMEs in capitalizing on supplier development opportunities in the oil and gas sector.

Other recent evaluation findings supported the notion that access to services across sub-programs increases the achievement of client outcomes. As reported in the IBD evaluation, 50% of all IBD clients said they received assistance under P&G and that this was a factor in the success of their project; and they rated the impact of their P&G project on SME outcomes as high (average rating: 4.08/5).

4.4 Barriers and Facilitators to the Achievement of P&G Outcomes

Judgment Criteria	Key Findings
Evidence that factors impeding the success of the P&G sub-program and lessons learned are known and mitigation strategies are implemented where appropriate. Evidence of factors facilitating the achievement of P&G expected outcomes, and the application and sharing of best practices where appropriate. Evidence that ACOA takes into consideration the findings/recommendations of previous evaluations /audits/reviews in improving delivery and considering alternative approaches related to P&G.	The most commonly cited facilitator for achieving outcomes was supporting the development of client skills and knowledge, for example through PBSI and other consulting and advisory services, which were considered by clients to have low costs and high returns. “Lean” productivity initiatives were considered a best practice, particularly in New Brunswick and Newfoundland and Labrador. Pro-active program promotion to SMEs by ACOA staff can increase program uptake. Non-financial advisory supports are highly valued by clients. Factors impeding the achievement of outcomes included economic conditions, SME financial strain, client knowledge gaps, and budgetary constraints on ACOA staff’s access to specialized knowledge development opportunities that equip them to be able to offer high quality advice to their clients. While ACOA is aware of barriers and has developed best practices in these areas, they also represent areas of on-going need. Most evaluation recommendations were fully implemented with the exception of the EBSD strategy, which was not implemented due to a strategic decision to pursue other priorities.

Judgment Criteria

Key Findings

Evidence that factors impeding the success of the P&G sub-program and lessons learned are known and mitigation strategies are implemented where appropriate.

Evidence of factors facilitating the achievement of P&G expected outcomes, and the application and sharing of best practices where appropriate.

Evidence that ACOA takes into consideration the findings/recommendations of previous evaluations /audits/reviews in improving delivery and considering alternative approaches related to P&G.

The most commonly cited facilitator for achieving outcomes was supporting the development of client skills and knowledge, for example through PBSI and other consulting and advisory services, which were considered by clients to have low costs and high returns. “Lean” productivity initiatives were considered a best practice, particularly in New Brunswick and Newfoundland and Labrador. Pro-active program promotion to SMEs by ACOA staff can increase program uptake. Non-financial advisory supports are highly valued by clients.

Factors impeding the achievement of outcomes included economic conditions, SME financial strain, client knowledge gaps, and budgetary constraints on ACOA staff’s access to specialized knowledge development opportunities that equip them to be able to offer high quality advice to their clients. While ACOA is aware of barriers and has developed best practices in these areas, they also represent areas of on-going need.

Most evaluation recommendations were fully implemented with the exception of the EBSD strategy, which was not implemented due to a strategic decision to pursue other priorities.

Barriers and Lessons Learned

Key stakeholders interviewed noted that difficult economic conditions during the evaluation period represented a barrier to achieving project outcomes. Client-related challenges that impeded project success were also raised by key informants. These included gaps in SME management skills that enable them to recognize threats and opportunities to increase productivity and/or profitability, such as specialized knowledge as it relates to their ability to adopt the right technology. Many stated that SMEs often lack the working capital required for growth, and some suggested that SMEs and funding partners lack awareness of the types of ACOA supports available. Stakeholders also noted that not-for-profits are sometimes challenged to build and maintain a level of expertise required to advise SMEs.

One lesson learned was that SME operators often fulfill many roles and have limited time and funding for training. As such, business skills development supports need to be practical, low-cost and targeted to the need of the business owner. Barriers that relate to internal factors were also noted, such as financial constraints associated with staff ability to access knowledge development activities (e.g., conferences and opportunities to access cutting-edge knowledge and key stakeholders pertaining to key sectors and market development). Key informants felt that such opportunities enable account managers to offer high quality advisory services and/or to direct SMEs to appropriate resources and suggested that ACOA find ways to ensure training needs are met, despite budget restrictions related to travel and conference attendance. Other internal challenges pertained to gaps in research and coordination in areas such as entrepreneurship development, emerging sectors and supplier development.

Approximately one-fifth of commercial client survey respondents (21%, n=52) identified factors that constrained project success. Most commonly identified factors included the recession (15%) and financial strain (15%), followed by project delays (9%) and attracting or retaining skilled workers (7%). The number of non-commercial respondents who answered this question was too small to report.

Many of these barriers and lessons learned are known and mitigated to some extent, by activities such as providing SMEs with access to expertise through PBSI, and consulting and advisory services. Key informants suggested other mitigations could be further developed, such as seeking out best practices between ACOA regions and outside the Atlantic region, harnessing the expertise of external subject matter experts to plan programming, as well as coordinating efforts and sharing knowledge between ACOA regions.

Facilitators and Best Practices

Key informants mentioned several factors that facilitate outcome achievement such as pro-active promotion of services to prospective SME clients by account managers and enhanced project planning, including the involvement of the right partners. The provision of timely access to expertise and consulting advice to SMEs was seen as a best practice that yielded high returns on investment. “Lean” productivity manufacturing initiatives undertaken in ACOA’s Newfoundland and Labrador and New Brunswick regions were also cited as best practices and were said to have resulted in tangible cost savings to SMEs. Others included seeking out best practices from similar organizations within Canada (e.g. other regional developments agencies) or internationally for possible implementation in the region, and entrepreneurship initiatives within post-secondary institutions that integrate entrepreneurship with other disciplines.

Over half of commercial client survey respondents (52%, n=115) identified factors that contributed to project success. Most commonly-identified factors included the hard work and dedication of ACOA staff to the project (20%) and talented staff and consultants (18%). Over two-thirds of non-commercial clients (68%, n=26) also identified contributing success factors such as obtaining provincial support (31%) and buy-in from industry partners and stakeholders (23%). The majority of clients surveyed (70%) stated that one of the reasons for applying to ACOA was the value of the non-financial supports (e.g., information and guidance) and 75% of those who offered comments at the end of the survey praised ACOA staff dedication and hard work, often crediting the support received with the successful growth or survival of their business. Internal key informants echoed the importance of non-financial supports and stated that given the increasing complexity of the business environment and sector development, it is important for account managers to continue to access learning and skills building opportunities as a means of supporting their role.

Many of these facilitators and best practices are known and broadly implemented, such as acquisition of consulting and expertise through PBSI or other mechanisms, the implementation of “lean” productivity initiatives, and enabling staff to access knowledge and capacity building opportunities.

Implementation of Recommendations from Previous Evaluations and Audits

Most recommendations from previous evaluations were addressed. Recommendations from the FC evaluation (2010) were, for the most part, fully implemented. The role of activities under the FC was further clarified and merged with other activities under P&G; however, logic model work is ongoing. A recommendation to improve service standards was addressed, with ongoing work in some areas.

As a result of a recommendation to develop an EBSD strategy, a deck was presented to ED directors and directors general of Operations Committee in 2011. Subsequent to the Business Skills Development evaluation, coordination with other areas was enhanced, and a broader framework was discussed in combination with the Atlantic Population Strategy in 2012. This work lost momentum due to increased focus on other Agency priorities such as efforts to increase innovation and productivity among growing SMEs.

The implementation of recommendations from audits completed by the Office of the Comptroller General in 2012 and 2015 is on-going. For example, in response to recommendations of the 2012 audit on provisionally repayable contributions, ACOA’s Special Accounts unit has developed training sessions related to contribution agreements, monitoring requirements and when to seek expert advice. The most recent receivables management audit (2015) recommended ACOA adapt project monitoring to changes in project risk. While consulted stakeholders indicated progress in these areas, the evaluation team has not verified the extent of implementation of audit recommendations.

5.0 Findings: Performance – Efficiency and Economy

The evaluation assessed efficiency and economy by examining the following: (1) evidence of the efficient utilization of resources, (2) the existence of mechanisms that promote efficiency, including those related to program delivery, coordination/collaboration and governance, and (3) the extent to which the program considers alternative delivery mechanisms.

5.1 Efficient Utilization of Resources

Judgment Criteria	Key Findings
Delivery costs are reasonable in relation to outcomes achieved.	Overall, P&G delivery costs have decreased over time, are reasonable when compared to other ED programming and present positive results in terms of both operational and allocative efficiency. ACOA’s cost to deliver $1 of G&Cs was $0.21 O&M over the six-year period. Compared to other evaluations, it is lower than both the 2010 FC Evaluation ($0.33), and 2015 IBD evaluation ($0.27) but higher than the 2015 IC evaluation ($0.09). The O&M as a percentage of total expenditures has decreased over time from 25.3% in 2008-2009 to 13.0% in 2013-2014, averaging 17.1% over the period, which has decreased since the 2010 FC Evaluation (24.6%). The decrease in O&M over the period also caused the O&M cost per project to decline; the six-year average being $21,060 per project, which is lower than IBD at $38,646 and IC at $49,733, and significantly lower than FC’s $73,278. A contributing factor to the decreasing cost is the increase in the total overall number of projects once FC and EBSD merged under P&G.

Judgment Criteria

Key Findings

Delivery costs are reasonable in relation to outcomes achieved.

Overall, P&G delivery costs have decreased over time, are reasonable when compared to other ED programming and present positive results in terms of both operational and allocative efficiency.

ACOA’s cost to deliver $1 of G&Cs was $0.21 O&M over the six-year period. Compared to other evaluations, it is lower than both the 2010 FC Evaluation ($0.33), and 2015 IBD evaluation ($0.27) but higher than the 2015 IC evaluation ($0.09).
The O&M as a percentage of total expenditures has decreased over time from 25.3% in 2008-2009 to 13.0% in 2013-2014, averaging 17.1% over the period, which has decreased since the 2010 FC Evaluation (24.6%).
The decrease in O&M over the period also caused the O&M cost per project to decline; the six-year average being $21,060 per project, which is lower than IBD at $38,646 and IC at $49,733, and significantly lower than FC’s $73,278. A contributing factor to the decreasing cost is the increase in the total overall number of projects once FC and EBSD merged under P&G.

Analysis of Program Delivery Costs

Overall, delivery costs have decreased over time and present positive results in terms of both operational[lx] and allocative[lxi] efficiency. The operational cost of delivering P&G programming was $0.21 O&M to deliver $1 G&Cs over the six-year period. The total decreased from $0.34 in 2008-2009 to $0.15 in 2013-2014. Compared to other evaluations, it is lower than both the 2010 FC Evaluation ($0.33), and 2015 IBD evaluation ($0.27). The 2015 IC evaluation reports a much lower ratio of $0.09, largely due to relatively low operating dollars and high amounts of G&C funding due to the Atlantic Innovation Fund. O&M costs per project have also declined over time; the six-year average is $21,060, compared to IBD at $38,646 and IC at $49,733, and is significantly lower than FC’s $73,278.[lxii]

Similarly, the O&M as a percentage of total expenditures has decreased over time from 25.3% in 2008-2009 to 13.0% in 2013-2014, averaging 17.1% over the period. This represents a significant decrease over the previous FC evaluation at 24.6%.

The decrease in delivery costs for P&G between 2008-2009 and 2013-2014 is largely due to both operating costs and salary dollars decreasing over time, mainly resulting from cross-government spending reductions, including a strategic review in 2010.[lxiii] FTEs have remained fairly stable, averaging 81.9 FTEs, with a slight increase in salary expenditures noted during 2012-2013 and 2013-2014, which is the result of the allocation of a portion of corporate salary expenditures to the P&G sub-program that supports P&G delivery.

For other operating costs, there has been a consistent downward trend in expenditures across all activities over the period, represented by a decrease of 54% in other operating costs between 2008-2009 and 2013-2014. Special/professional services represented the highest operating activities funded across all regions and HO at 60.2%, (a low of 35.4% at HO to a high of 68.3% in Nova Scotia[lxiv]) followed by transportation/communications at 21.89% (a low of 12.2% in Nova Scotia to a high of 60.9% at HO).

The client survey identifies a number of measures that support allocative efficiency, or costs associated with achieving certain program outcomes. The survey results allow for the estimation of average financial impacts for clients[lxv] (e.g., cost savings from productivity gains or new investments attracted) to calculate the return for every dollar of funding approved by ACOA. The results show that clients are seeing positive impacts in terms of improvements to productivity and efficiency of operations and increased revenues as a result of ACOA funding. These are presented in Table 9.

Table 9: Reported Impacts Resulting from Project per Dollar of ACOA Funding Provided

Client Survey Impacts Per $ of ACOA Funding Provided
Improvement to Productivity and Efficiency of Operations (n=88)
Savings as a result of ACOA supported projects	$13,233,500
ACOA funding	$11,608,768
Savings per dollar of ACOA funding (n=88)	$1.14
Increased Revenues – Latest Fiscal Year (n=85)
Last year ($ M)	$60,173,837
ACOA funding ($M)	$14,463,322
Revenues per dollar of ACOA funding	$4.16
Increased Revenues –Current Year (n=86)
Expected current year ($M)	$59,575,260
ACOA funding ($M)	$16,135,494
Revenues per dollar of ACOA funding	$3.69

Source: Client Survey, May 2015

While there are no previous results that can be used for comparison, these impact measures present a baseline that can assist data analysis and comparison in future evaluations.

5.2 Mechanisms Supporting Efficiency and Economy

Judgment Criteria	Key Findings
ACOA has in place resource optimization mechanisms to ensure that the most efficient and economical means are used to administer the programming. To what extent has program management considered and implemented alternative modes of delivery for the P&G sub-program?	ACOA has a number of mechanisms in place to support the efficient and economical delivery of programming, particularly those related to: a decentralized delivery model, internal communication/coordination, resource leveraging, and performance measurement. Others include the use of governance mechanisms and process/tool enhancements to improve delivery. There is an opportunity to enhance efficiency and economy of the P&G programming by making further improvements to processes and procedures related to: program delivery coordination, collaboration and governance ACOA’s program management has considered and continues to explore alternative modes of delivery to some extent.

Judgment Criteria

Key Findings

ACOA has in place resource optimization mechanisms to ensure that the most efficient and economical means are used to administer the programming.

To what extent has program management considered and implemented alternative modes of delivery for the P&G sub-program?

ACOA has a number of mechanisms in place to support the efficient and economical delivery of programming, particularly those related to: a decentralized delivery model, internal communication/coordination, resource leveraging, and performance measurement. Others include the use of governance mechanisms and process/tool enhancements to improve delivery.

There is an opportunity to enhance efficiency and economy of the P&G programming by making further improvements to processes and procedures related to:

program delivery
coordination, collaboration and governance

ACOA’s program management has considered and continues to explore alternative modes of delivery to some extent.

5.2.1 Factors Contributing to Efficiency and Economy

Many internal key informants indicated that P&G programming is being delivered in an efficient and economical manner and identified a number of mechanisms that support efficiency and economy. These include the existence of working groups/committees who regularly discuss improvements to program delivery, the utilization of web technology to deliver training, enhancements to processes and forms related to contribution agreements and the application process, and ongoing technology/tool enhancements such as the automation of some functions in QAccess and promotion of the use of SharePoint. Those that stood out, related more specifically to the delivery model, resource leveraging and performance measurement.

Delivery Model

As noted in recent evaluations of other ED programming, many aspects of P&G’s program design and delivery promote efficiency and economy. The decentralized and flexible nature of the programming allows the Agency to undertake a client-centred approach with local businesses and stakeholders (e.g. provincial government) to develop relationships and respond to regional needs and opportunities. Key informants indicated that a decentralized delivery model also allows program staff to focus their efforts on key regional sectors to maximize impacts while increasing program reach through not-for-profits that support SMEs (i.e., sector associations and EBSD projects).

Resource Leveraging

As noted in Section 4.1, the efficiency and economy of P&G resources is increased through leveraging with key partners. Leveraging per ACOA dollar invested was positive for commercial repayable projects ($2.13) but was much less for PBSI ($0.60), largely due to the smaller nature of these projects, where ACOA is often the only funder besides the client. Non-commercial projects leveraged $1.38 from other sources, up from the previous evaluations ($1.18 EBSD; $0.80 FC).

Performance Measurement

The evaluation found that while there was little change in how performance measurement was undertaken over the period of the study, program managers are aware of its importance to measuring and reporting on P&G activities. There are currently a number of activities underway to strengthen the approach to measuring and monitoring P&G performance while also considering its alignment to other sub-programs. Activities related to updating the P&G logic model and the use of dashboards for monitoring project performance are currently underway and are expected to contribute to increased effectiveness and efficiency moving forward.

5.2.2 Opportunities for Improving Efficiency and Economy

Areas that represent potential opportunities for improvement include those related to program delivery, coordination, collaboration and governance, as well as alternative delivery mechanisms. These opportunities for improvement go beyond P&G, having a potential impact on all sub-programs. These findings are further supported by previous evaluations completed across sub-programs.

Program Delivery

Elements directly related to program delivery that present potential areas for improvement are directly related to BDP policy (which includes PBSI), processes and procedures, delegated authority limits, as well as the approval of larger projects (over $500,000) and relationships with the Special Accounts Unit at HO.

BDP policy and related procedures:

At the request of senior management, the evaluation inquired about the degree to which parameters (e.g., funding approval limits) around delivery of BDP/PBSI projects are appropriate and are meeting the needs for effective program delivery. The BDP transfer payment program has been in place since 1996 with the manual outlining the policies and procedures available to staff through the Agency’s intranet. Key informants reported that while parts of the policy are updated on an ongoing basis, portions of the BDP policy relating to certain eligibility criteria may benefit from further consideration and may not be reflective of the changing economic and business landscape. Key informants suggested ACOA could explore whether the wording and/or interpretation of sector eligibility criteria should evolve to enable the Agency to address opportunities to increase competitiveness in some sectors that are important to the region, and account for evolving realities and needs. For example, staff interviewed expressed a need to further discuss support to the business services sector, citing that although guidelines allow for project support under certain circumstances, there is a need further discuss these conditions and/or to better understand them. A second concern raised by key informants related to the level of detail contained in project summary forms (PSF), which aims to summarize key aspects of a project and the rationale for providing support. Key informants indicated that PSFs often contain a significant amount of detail that may not be necessary, leading to longer preparation and review times. Key informants also suggested that PSF requirements could be better calibrated to user needs, or project and portfolio risk, creating the opportunity for streamlining.
PBSI limit and eligibility requirements:

Both key informants and client survey respondents indicated that PBSI (funded through BDP) is a highly valued and successful programming tool that produces high returns on investment. Key informants indicated that the PBSI eligibility requirements and lifetime limit of $100,000 per business may present barriers to addressing the needs of businesses which evolve over the longer-term. For example, the policy manual states that companies are “not normally” eligible for more than $100,000, and that “normally, they would be beyond the start-up stage”. Regions have interpreted eligibility requirements and limits differently, resulting in inconsistent application across regions. Some regions have interpreted this wording as rarely or never employing PBSI to support start-ups, while other regions use it as a tool primarily to support start-ups as a means of increasing SME capacity for innovation/commercialization.
Delegated authority limits:

According to the TB Directive on Delegation of Financial Authorities for Disbursements, the responsible and risk-based delegation of functional authorities contributes to sound management and is a key internal control mechanism. The appropriate delegation of authorities can empower the employees of the organization and permit their development and growth while maintaining accountability.[lxvi] A number of key informants questioned the degree to which the regional delegated authority limits of $500,000[lxvii] continue to be appropriate given the increasing cost to do business, particularly for technology and equipment purchases, while also taking into consideration the level of risk of the project/portfolio. Though delegation of authority for various approval levels vary by region, ACOA’s $500,000 delegated authority limit is consistent across regions and has been in place since the establishment of the BDP in 1996.
Approval of Projects over $500,000:

Directly linked to delegated authority limits are the processes surrounding the approval of projects over $500,000. All projects are reviewed by the Internal Review Committee (IRC), a pan-Atlantic ACOA committee of senior management. The IRC review is valued by management as it brings added due diligence and accountability on larger projects and assesses competitive impact on an Atlantic scale. According to key informants, while the turn-around time from the IRC committee is quick (i.e., approximately one week), the administrative processes leading up to and after the IRC review adds a number of weeks to the project approval timeline. Internal interviewees felt that the IRC review adds value but felt that the extra resources and time required by the processes surrounding the review can act as a disincentive to submit projects to the process, and result in a substantial number of projects being approved at, and just under the $500,000 threshold, with some key informants stating that they avoid delivering projects exceeding that amount. Others indicated that because of the delays caused by processes, the program is often promoted as a program with a maximum funding limit of $500,000, even though authority exists to approve projects up to $10 million. Data shows that 5% of P&G projects (135 projects), representing 20% of funding ($67.5 million), were funded at exactly $500,000, with another 1% of projects (31 projects), representing 5% of funding ($15 million) approved between $450,000 and $499,000. Only 1% of P&G projects (29 projects), representing 11% of funding ($36.9 million), were approved above $500,000 during the period of the study.
Special Accounts (SAs):

The SAs unit is located at head office and provides consultation services to the regions regarding the administration of the Agency’s accounts receivable portfolio and performs collection measures/procedures regarding defaulted grants and contribution agreements in order to maximize recovery and protect the Crown’s overall interest. Key informants indicated that the SAs unit staff may not be engaged by program staff in a timely manner to help with complex files experiencing specific challenges, resulting in missed opportunities to use the knowledge and experience of the SAs unit to assist clients or to collect funds owed to the Agency. Ensuring that staff know how and when to engage the SAs reduces the risk for the Agency in terms of recovery of funds, especially since the statute of limitations[lxviii] to undertake civil proceedings has decreased to two years in some regions of the country, with all other regions expected to follow suit. This risk is further impacted by the increasingly complex nature of projects/portfolios being undertaken by the Agency. This is consistent with the findings of previous audits,[lxix] which highlight the importance of training sessions provided to staff by the recovery unit, and the importance of the SAs’ role in effective monitoring and management of project/portfolio risk. These audits recommended that program procedures be updated to include when staff should seek advice from the recovery unit. While SA training sessions have been recognized as a best practice by the OCG, interviewed ACOA staff indicate that opportunities exist for better engagement between the SAs and program staff and for additional training to build the knowledge of program staff related to specific SME issues and when to seek advice from the SAs unit.

Coordination, Collaboration and Governance

The evaluation highlighted a number of mechanisms related to coordination, collaboration and governance that support effective and efficient program delivery.

ACOA adopted a more integrated approach to programming in 2010 when the FC and EBSD sub-programs were merged to create the P&G sub-program. Key informants reported that internal coordination is generally considered effective, with a strong linkage between P&G and IBD in terms of growth, increasing productivity, and expanding markets internationally. Key informants raised the need for greater coordination and collaboration on a pan-Atlantic basis, particularly between ED and CD related to entrepreneurship and business skills programming, where activities undertaken under P&G are similar in nature to projects supported under Community Investment (CI) programming. (Average CI funding is $3.4 million approved/year between 2009-2010 and 2013-2014, as reported in the Community Mobilization and Community Investment evaluation). Internal key informants raised the importance of capturing and reporting these activities as a component of these projects that support entrepreneurship and the development of business skills when considering, undertaking, planning, delivering, and reporting on activities, which will contribute to providing a more holistic view of ACOA’s support for these activities. Key informants indicated that improvements to coordination and collaboration could be aided by greater coordination and collaboration between regions and HO in key areas such as entrepreneurship and business skills development, supplier development or other areas of need.

As previously mentioned, ACOA has a well-established governance structure, where P&G is delivered in a decentralized manner across four regions, supported by the Director General, Enterprise Development at HO. Key informants highlighted the importance of this delivery structure as it allows the programming to be implemented in a way that can be tailored to reflect region-specific priorities, needs and context, while facilitating a strong client-centered approach to program delivery.

Similar to other sub-programs, planning strengths are evident in that agency priorities for P&G programming are identified corporately, and regional plans are articulated in annual integrated planning documents.

Alternative Approaches to Delivery

ACOA’s program management considers and explores alternative approaches to P&G delivery to some extent. In recent years, the Nova Scotia regional office has put a greater focus on rural areas by proactively targeting SMEs in those areas. Support for rural areas is also evident through ongoing collaboration with other organizations (e.g. CBDCs and other region-specific organizations) that support start-ups and/or rural SMEs The decentralized approach to delivery enables different regions to focus on sectors that are considered a priority in their area. For example, in Nova Scotia and New Brunswick there is a focus on aquaculture; New Brunswick and Newfoundland and Labrador have identified manufacturing lean as a key initiative; Prince Edward Island and Newfoundland and Labrador have a focus on supply chain development; and Prince Edward Island has a focus on bioscience. Given the variety of sector focus across regions, key informants felt that this presents an opportunity to enhance the sharing of knowledge, best practices and lessons learned. Key informants felt an opportunity exists to contribute to greater information/experience sharing with other regional development agencies, which could broaden the options available to ACOA in terms of alternative delivery mechanisms.

6.0 Conclusions and Recommendations

The P&G evaluation conclusions and recommendations were identified based on thorough analysis of the key findings presented in this report. They were validated through discussion and consultation with the Evaluation Advisory Committee, ACOA senior management and other stakeholders.

6.1 Conclusions

Relevance:

Key informants identified opportunities for greater collaboration and consultation with external stakeholders and experts to discuss broader policy issues and to gain access to research in support of decision making, including policy research for modifying programs, setting strategy, and strategic intelligence for working with clients.

Atlantic Canada’s aging population, which is the oldest in Canada, combined with the outmigration of youth have exacerbated the need for youth entrepreneurship promotion and supports, as well as the need for business succession planning to increase the sustainability of viable businesses as owners retire. As the region faces challenges with a shrinking labour force, including those who are self-employed and other types of labour (skilled and unskilled), there is also a need to increase entrepreneurship for under-represented groups such as women. During the evaluation period, entrepreneurship and business skills funding programs aimed at youth and women were folded into the Agency’s larger BDP. ACOA continued to support initiatives aimed at women with some decline in youth entrepreneurship programming under P&G programming. Internal key informants expressed a desire to increase the Agency’s focus on entrepreneurship, as well as coordination and information sharing across regions, which has declined in recent years. While ACOA has recently taken steps to explore the further use of GBA+ within the Agency, it has not been decided how GBA+ will be further incorporated into policy or planning activities.

As a complement to annual planning exercises, program stakeholders raised the need for an articulated and communicated overarching, longer term strategy for P&G that reflects key objectives, activities and priorities, covering commercial and non-commercial entrepreneurship and business skills supports (e.g., general and/or aimed at specific groups such as youth and women, sector associations, etc.).

Effectiveness: Overall, the evaluation found that the Agency’s P&G activities have had a positive incremental impact on the implementation of projects, leveraging of funds and achievement of outcomes. The achievement of P&G outcomes was supported by an increase in repayable contributions approved over time. Areas of outcomes achievement for both repayable contributions and PBSI clients include increased production volume, improvements in product quality and productivity, competitive position, and employment. Notably, PBSI results also include the development of management and marketing skills. Areas for continued development and focus include productivity and the development of new markets.

Efficiency and Economy: Delivery costs have declined and compare well to other program areas. A number of mechanisms are in place related to design and delivery that contribute to the effective and efficient use of resources, including: decentralized delivery and flexible programming that is able to respond to needs at a regional level, focus in key sectors to maximize impacts and increase the reach of SMEs through not-for-profits resource leveraging, performance measurement, established governance structures, internal communication/coordination mechanisms, as well as consideration of alternative approaches to delivery.

Some existing and potential inefficiencies pertain to specific BDP policies and procedures that go beyond P&G, particularly those related to funding limits and some eligibility requirements. Areas where efficiency and economy could be enhanced are through improvements to policies and procedures related to program delivery, communication and coordination on a pan-Atlantic basis and/or with external stakeholders, as weaknesses in these areas pose potential risks to the efficient and economical use of resources and achievement of outcomes. In terms of capacity building, staff require access to specialized knowledge and skills to be able to provide the advice necessary to efficiently address the growing number of complex SME issues.

6.2 Recommendations

The following two evaluation recommendations have been discussed with the members of the EAC and ACOA senior management. Each recommendation aims to build upon progress made since the previous evaluation, while ensuring that ACOA takes a strategic approach to programming even as it considers emerging programming needs on an ongoing basis. This is to be accomplished by developing a long-term strategic vision for P&G programming through the development of a P&G framework and by enhancements in the efficient utilization of resources by improving upon processes and procedures impacting program delivery.

Table 10 presents the linkages among findings, conclusions and recommendations.

Recommendation 1: In light of the evolving demographics of entrepreneurs and the increasingly complex and global nature of business, it is timely for ACOA to build on its current policy research and knowledge base, best practices and other planning exercises to develop and communicate an overall long-term Productivity and Growth Framework that will promote a clear corporate direction for programming while allowing for regional flexibility.

Given the programming’s decentralized delivery and regional differences in context and needs, it is recommended that the framework be developed collaboratively, updated regularly and promoted as a strategic planning and internal communication tool, as a complement to the annual planning exercises.

The following is a list of considerations in developing the framework that build on evaluation findings:

the positioning of P&G among other ACOA supports and an increasingly complex landscape of external programming;
key activities and initiatives, including P&G repayable contributions to certain sectors, PBSI and non-commercial programming as it relates to entrepreneurship and business skills development for certain sectors or groups (Note: identification of key activities and desired outcomes can build upon on-going logic model/performance measurement review; and planned Entrepreneurship and Business Skills Review);
regional P&G context and priorities and their key role in delivery (e.g. key sectors / areas of focus for commercial and non-commercial supports);
effective consultation and collaboration approaches with key external pan-Atlantic and regional stakeholders (collaborators, clients, beneficiaries, and others) as well as collaborators outside the region, which support on-going relevance and success of P&G; and,
governance and other mechanisms that support internal planning, communication and coordination within and between regions, including sharing best practices and capitalizing on available information to inform programs.

In keeping with the findings of this evaluation, it is advised that management explore the following:

review eligible sectors and activities supported by the BDP to determine whether changes are required to reflect the evolving business landscape/SME needs;
aim to strike a balance between efficiency and oversight/control in adopting a risk based approach to project delivery processes, including streamlining PSF content and reviewing approval processes for large projects (>$500,000);
review PBSI guidelines related to the total funding available over the SME’s lifespan and to its use for start-up SMEs;
continue to build account manager knowledge and capabilities to support them in navigating the increasing complexity of business growth and business supports. This will assist with continued strong support to SMEs and ensure stewardship of public funds.

Table 10: Alignment of P&G Evaluation Findings, Conclusions and Recommendations

Summary of Key Findings	Summary of Conclusions
Relevance	P&G programming is aimed at addressing most areas of demonstrable need but challenges exist with accessing policy research and strategic intelligence to address some evolving needs. There is a desire to for greater consultation with external stakeholders/ experts to support policy and program decision-making in some areas. Aging population/business owners, youth outmigration, and the shrinking labour market have increased the need for entrepreneurship and business skills supports to youth, and other under-represented groups such as women. During the evaluation, P&G entrepreneurship and business skills initiatives aimed at youth and women were folded into the larger BDP, which resulted in some decline in these supports to youth. There is a desire to increase coordination and focus in this area. It was noted that entrepreneurship activities are supported under other areas of the Agency. Though annual planning exercises are undertaken, the absence of an articulated and communicated P&G programming framework that explains its key activities was noted. P&G programming aligns well with federal government priorities, roles and responsibilities, and is complementary to that of other funders, but is part of a complex landscape where awareness and understanding of programming is unclear. The programming has had a positive incremental effect on the implementation of projects, leverage of funds, and achievement of outcomes, particularly with respect to productivity improvements, increases in volume/quality of products produced, increases in sales and competitive position, and the development of business skills. Areas for continued development and focus include productivity and the development of sales and new markets. The majority of repayable contributions due over this timeframe were paid back to the Agency. Factors such as the development of Account Manager and client skills/knowledge are important for achieving program outcomes. Pro-active program promotion to SMEs by ACOA staff facilitates program delivery. While ACOA is aware of these factors and has developed best practices in these areas, they also represent areas of on-going need. Clients’ value non-financial supports (e.g. advice/guidance) received. Delivery costs have declined and compare well to other program areas. ACOA’s P&G delivery model demonstrates a number of efficiencies such as the flexibility to respond to regional needs, maximize impacts in key sectors; and increased reach of SMEs through not-for-profits. A number of internal measures have been initiated to improve efficiencies. New dashboards are expected to create greater efficiencies in monitoring, while the logic model and related performance measurement framework is being revised. Some inefficiencies pertain to certain policies and procedures that go beyond P&G, particularly those related to funding limits and eligibility requirements. Opportunities exist for greater coordination and collaboration with other ACOA sub-programs on entrepreneurship projects, and on a pan-Atlantic scale and/or with external stakeholders on various P&G initiatives, Capacity gaps that exist for program staff contribute to challenges in addressing some complex SME issues.	Recommendations Develop a P&G Framework (Links to conclusions (1,2,3,4,5) 2. Improvements to policies and procedures (Links to conclusion 6) Strategic Policy Framework
Since the previous evaluation, continued and growing areas of challenge for businesses include SME access to capital, productivity and SME knowledge/skills (e.g. management, market development and sales, entrepreneurship, succession planning and business take-up). P&G activities aim to address most needs to some extent.
The need for youth entrepreneurship supports has increased, ACOA’s funding for these types of projects under P&G have decreased. Some youth entrepreneurship projects are also supported under other ACOA sub-programs. Support for business support projects aimed at women entrepreneurs has remained consistent. The need for this programming has remained consistent as well (women are under-represented as business owners). Stakeholders expressed a desire to increase planning and coordination of entrepreneurship and business skills programming aimed at these groups to mitigate the risk of a decrease in focus.
Key informants identified opportunities for greater collaboration and consultation with external stakeholders and experts to discuss broader policy issues, and to gain access to strategic intelligence in support of decision-making on more specific issues, whether at a program or project level. While important policy research has taken place to support decision-making, certain information gaps may require new research. Key informants noted the absence of an articulated and communicated P&G strategy that reflects key objectives, activities and priorities.
ACOA’s role in P&G programming is appropriate and aligns well with federal government priorities and strategies. The landscape of funding programs is complex and difficult for SMEs to understand. Other funders/stakeholders report gaps in knowledge of ACOA programs. Programming is considered complementary; there do not appear to be large areas of overlap.
Effectiveness
P&G has had an incremental effect on clients’ abilities to proceed with their projects, the quality, scope and timeliness of projects, and leverage of funds.
P&G outcomes achieved for all SME clients include: increases in amounts of repayable contributions approved over time, increased production capacity, improvements in product quality and productivity, competitive position, and employment. Notable PBSI results also include the development of management and marketing skills. ACOA-supported SMEs out-performed unassisted SMEs in sales and labour productivity growth, and in survival rates. Amount of contributions repaid as a percentage of total receivables was 84%. Non-profit organizations that supported businesses reported increasing SMEs market knowledge, marketing skills and activities and strategic partnerships. The majority of beneficiaries surveyed reported increasing business skills. ACOA staff interviewed suggest further increasing development of sales, new markets, productivity and entrepreneurship.
Facilitators of success included the proactive promotion of programming, account manager knowledge development, and the involvement of key partners in project planning. Advice to SMEs is highly valued. Barriers to success included negative economic conditions, SME financial strain, clients’ lack of business skills and knowledge, and financial constraints around ACOA staff access to knowledge and skills development for continued service to businesses.
Most recommendations of previous evaluations were implemented; work continues on logic-models, performance measurement, and service standards.
Efficiency and Economy
Cost of delivery of 1$ of G&C was $0.21. Costs decreased since previous evaluations and compare well to other ED sub-programs.
Aspects of program design promote efficiency such as flexibility to respond to regional needs; focus on key sectors and increased reach of SMEs through non-profits. Internal coordination is effective within ED, but opportunities exist to enhance tracking/linkage of entrepreneurship and business skills development activities being undertaken by CD. Key informants expressed a desire for greater inter-regional coordination and collaboration on key files.
Inefficiencies raised pertain to certain policies and procedures such as calibrating procedures to project risk; approval of large projects (<$500K); PBSI parameters; the need to review eligible sectors/activities; and, challenges in dealing with complex SME issues.
ACOA has recently initiated a number of activities aimed at strengthening its approach to monitoring P&G performance, and continues work on the logic-model and outcome measurement.

Appendix A – P&G Evaluation Management Action Plan

Recommendations	Management Responses	Planned Actions	Responsibility	Target Date
Recommendation 1: In light of the evolving demographics of entrepreneurs, and the increasingly complex and global nature of business, it is timely for ACOA to build on its current policy research and knowledge base, best practices, and other planning exercises to develop and communicate an overall long-term Productivity and Growth Framework that will promote a clear corporate direction for programming, while allowing for regional flexibility. Given the programming’s decentralized delivery and regional differences in context and needs, it is recommended that the framework be developed collaboratively, updated regularly, and promoted as a strategic planning and internal communication tool, as a complement to the annual planning exercises. The following is a list of considerations in developing the framework that build on evaluation findings: The positioning of P&G among other ACOA supports and an increasingly complex landscape of external programming; Key activities and initiatives, including P&G repayable contributions to certain sectors, PBSI and non-commercial programming as it relates to entrepreneurship and business skills development for certain sectors or groups (Note: identification of key activities and desired outcomes can build upon on-going logic model/performance measurement review; and planned Entrepreneurship and Business Skills Review); Regional P&G context and priorities and their key role in delivery (e.g. key sectors / areas of focus for commercial and non-commercial supports); Effective consultation and collaboration approaches with key external pan-Atlantic and regional stakeholders (collaborators, clients, beneficiaries, and others) as well as collaborators outside the region, which support on-going relevance and success of P&G; and, Governance and other mechanisms that support internal planning, communication and coordination within and between regions, including sharing best practices and capitalizing on available information to inform programs.	Agree	ACOA will develop a long-term P&G Framework, building on its existing body of policy research and corporate knowledge, as well as undertaking further research and analysis as required. This framework will be developed through a coordinated approach between head office and the regional offices, with stakeholder engagement. The framework will identify key areas of focus, priorities and activities for the P&G program sub-activity, incorporating regional office context and priorities. More specifically, the framework will present a strategic approach for : the Agency’s direct support to SMEs within the P&G program activity, in the context of all Enterprise Development direct investments; and initiatives related to entrepreneurship and business skills development, considering the Agency’s overall activities in the area of skills development. This framework will be developed collaboratively to establish a corporate approach which will guide the Agency’s investments and programming for the Agency’s Productivity and Growth program sub activity. The development of the P&G framework will be done in parallel with the development of a logic model for the P&G sub-activity to ensure consistency and alignment in terms of key activities, priorities and outcomes. To ensure effective implementation and monitoring of the framework, ACOA will develop a communication plan that supports engagement and coordination, including a process to monitor the strategy to ensure it that it is achieving the desired outcomes.	Head Office Director General, Enterprise Development, in consultation with Regional and Head Office Directors, Enterprise Development and Head Office Policy	September 30, 2016
Recommendation 2: Building on the efficiency and effectiveness of P&G programming, it is recommended that ACOA explore and pursue opportunities for improvements related to delivery processes, recognizing that these have implications that reach beyond P&G programming. Existing governance structures should be used to prioritize actions in the context of available resources, existing plans and priorities, as well as relative risks to efficiency and effectiveness. In keeping with the findings of this evaluation, it is advised that management explore the following: Review eligible sectors and activities supported by the Business Development Program (BDP) to determine whether changes are required to reflect the evolving business landscape/small and medium-sized enterprise (SME) needs; Aim to strike a balance between efficiency and oversight/control in adopting a risk based approach to project delivery processes, including streamlining Project Summary Form content and reviewing approval processes for large projects (>500K); Review PBSI guidelines related to the total funding available over the SME’s lifespan and to its use for start-up SMEs; Continue to build account manager knowledge and capabilities to support them in navigating the increasing complexity of business growth and business supports. This will help assist with continued strong support to SMEs and ensure stewardship of public funds.	Agree	The Agency recognizes the importance of efficient and effective program delivery, and has taken a collaborative approach to identifying and addressing issues that impact both internal and external stakeholders. Recent examples of such initiatives include the development of Operational Dashboards to assist program delivery staff in identifying and prioritizing portfolio management issues, and a streamlining of the Agency’s claims payment processes. ACOA will continue to explore and pursue opportunities for improvements in delivery processes by initiating a process to identify and prioritize improvements to processes, leveraging expertise in head office and the regions and making use of existing governance structures including the Director Generals Operation Committee. The specific issues raised in this evaluation will be considered in the context of this prioritization exercise. Results of this continuous improvement initiative will be reviewed by the president and the executive committee on an annual basis.	Director General, Enterprise Development in collaboration with regional and head office directors general.	Monitoring and reporting on continuous improvement exercises will occur over a two-year period ending September 30, 2017.

Recommendations

Management Responses

Planned Actions

Responsibility

Target Date

The positioning of P&G among other ACOA supports and an increasingly complex landscape of external programming;
Key activities and initiatives, including P&G repayable contributions to certain sectors, PBSI and non-commercial programming as it relates to entrepreneurship and business skills development for certain sectors or groups (Note: identification of key activities and desired outcomes can build upon on-going logic model/performance measurement review; and planned Entrepreneurship and Business Skills Review);
Regional P&G context and priorities and their key role in delivery (e.g. key sectors / areas of focus for commercial and non-commercial supports);
Effective consultation and collaboration approaches with key external pan-Atlantic and regional stakeholders (collaborators, clients, beneficiaries, and others) as well as collaborators outside the region, which support on-going relevance and success of P&G; and,
Governance and other mechanisms that support internal planning, communication and coordination within and between regions, including sharing best practices and capitalizing on available information to inform programs.

Agree

ACOA will develop a long-term P&G Framework, building on its existing body of policy research and corporate knowledge, as well as undertaking further research and analysis as required. This framework will be developed through a coordinated approach between head office and the regional offices, with stakeholder engagement.

The framework will identify key areas of focus, priorities and activities for the P&G program sub-activity, incorporating regional office context and priorities. More specifically, the framework will present a strategic approach for :

the Agency’s direct support to SMEs within the P&G program activity, in the context of all Enterprise Development direct investments; and
initiatives related to entrepreneurship and business skills development, considering the Agency’s overall activities in the area of skills development.

This framework will be developed collaboratively to establish a corporate approach which will guide the Agency’s investments and programming for the Agency’s Productivity and Growth program sub activity.

The development of the P&G framework will be done in parallel with the development of a logic model for the P&G sub-activity to ensure consistency and alignment in terms of key activities, priorities and outcomes.

To ensure effective implementation and monitoring of the framework, ACOA will develop a communication plan that supports engagement and coordination, including a process to monitor the strategy to ensure it that it is achieving the desired outcomes.

Head Office Director General, Enterprise Development, in consultation with Regional and Head Office Directors, Enterprise Development and Head Office Policy

September 30, 2016

In keeping with the findings of this evaluation, it is advised that management explore the following:

Review eligible sectors and activities supported by the Business Development Program (BDP) to determine whether changes are required to reflect the evolving business landscape/small and medium-sized enterprise (SME) needs;
Aim to strike a balance between efficiency and oversight/control in adopting a risk based approach to project delivery processes, including streamlining Project Summary Form content and reviewing approval processes for large projects (>500K);
Review PBSI guidelines related to the total funding available over the SME’s lifespan and to its use for start-up SMEs;
Continue to build account manager knowledge and capabilities to support them in navigating the increasing complexity of business growth and business supports. This will help assist with continued strong support to SMEs and ensure stewardship of public funds.

Agree

The Agency recognizes the importance of efficient and effective program delivery, and has taken a collaborative approach to identifying and addressing issues that impact both internal and external stakeholders. Recent examples of such initiatives include the development of Operational Dashboards to assist program delivery staff in identifying and prioritizing portfolio management issues, and a streamlining of the Agency’s claims payment processes.

ACOA will continue to explore and pursue opportunities for improvements in delivery processes by initiating a process to identify and prioritize improvements to processes, leveraging expertise in head office and the regions and making use of existing governance structures including the Director Generals Operation Committee. The specific issues raised in this evaluation will be considered in the context of this prioritization exercise.

Results of this continuous improvement initiative will be reviewed by the president and the executive committee on an annual basis.

Director General, Enterprise Development in collaboration with regional and head office directors general.

Monitoring and reporting on continuous improvement exercises will occur over a two-year period ending September 30, 2017.

Appendix B – P&G Evaluation Framework: Questions, Judgment Criteria and Methods

Core Evaluation Issues and Key Evaluation Questions		Proposed Evaluation Methodology						Level of effort
Core Evaluation Issues and Key Evaluation Questions		Doc. Review	Interview (AM, M, E/S)	Client survey	Administrative Data
Proposed P&G Evaluation Question	Evaluation Judgment Criteria [lxx]	Doc. Review	Interview (AM, M, E/S)	Client survey	QA	GX	PM data
Issue 1: Continued need for programming
1.1 To what extent does the P&G sub-program continue to address demonstrable needs? Are there gaps in services needed to assist businesses with respect to P&G programming?	Programming related needs are still present at least to the same degree as they were five years ago. Evidence that the needs for financial assistance for P&G activities are the same as 5 years ago, have been maintained or have increased, including those related to: promoting entrepreneurship; developing business skills; accessing required expertise obtaining industry certifications establish a business (both traditional establishments and technology-oriented start-ups) expand/modernize a business (e.g. expanding operation, increasing productivity,) conducting domestic marketing activities. Note: More specific needs or challenges may relate to: speed of business, access to skilled labour, global nature of business / supply chains, maintaining competitiveness, succession planning, needs of growth vs lifestyle firms. Information will also be gathered on suggested types of assistance (e.g. contributions, non-repayable). Information on the changing SME landscape in Atlantic Canada (i.e. demographic, economical, sectorial, strategic, other programming) is considered in decision making by key stakeholders (i.e. ACOA mgmt.). Evidence of appropriate (necessary and effective) programming responses to changing landscape and needs, including changes in other programming.	X	X (AM, M, E/S)	X	X			Low
1.2 What has been the nature and extent of EBSD programming targeted to youth and women over the 6-year period? Should there be programming strategies around the targeting of youth and women or can they be well served with a general approach to programming?	Evidence on the nature and extent of ACOA support to youth and women and its evolution over the evaluation period. Evidence on the need for targeted programming strategies to support women and youth, and on maintaining a general programming approach.	X	(AM, M, E/S)		X			Med
1.3 Do certain parameters around delivery of P&G projects continue to address needs and allow for effective program delivery (i.e. funding approval limits under BDP and PBSI limits)?	Evidence that the BDP funding approval limits are meeting the needs of clients and allow for effective delivery of P&G programming. Evidence that PBSI funding limits (cap of $100,000 and maximum of 2 projects per client) are appropriate, are meeting the needs of clients, and allow for effective program delivery.	X	X (AM, M)	X	X			High
Issue 2: Alignment with Government Priorities
2.1 To what extent does the P&G sub-program align with and help deliver on federal government priorities and to ACOA’s strategic outcome.	There is logical alignment between the programming, federal government priorities (e.g. Speeches from Throne) and ACOA’s strategic outcome, priorities and strategy. The alignment is recognized, communicated and/or made explicit.	X						Low
Issue 3: Alignment with Federal Roles and Responsibilities
3.1 To what extent does the P&G sub-program align with federal roles and responsibilities?	P&G programming is aligned with the ACOA’s roles and responsibilities as outlined in the ACOA Act. ACOA’s P&G programming is aligned with policy research and current policy thinking on roles and responsibilities of federal government in the delivery of P&G types of programming (e.g. considering OECD, programming in other jurisdictions, etc.)	X	X (E/S)					Low
3.2 To what extent does P&G sub-program activities duplicate, overlap, or complement other programming, including other ACOA sub-programs?	ACOA P&G programming complements (rather than duplicates or overlaps with) other programming, including other federal and provincial governments. There is evidence of complementarity and coordination of P&G programming with IC and IBD; opportunities for improved coordination are known and being acted upon.	X	X (AM, M, E/S)					Low
Issue 4: Achievement of Expected Outcomes
4.1 Incrementality: What impact would the absence of P&G programming have on expected program outcomes?	The impact of the absence of the sub program is similar or greater to that observed in the previous evaluation, as reflected by: ACOA clients’ perspective on whether the project/initiative would have proceeded without funding. ACOA clients’ perspective on the impacts that would have occurred if there was not ACOA funding. Evidence that ACOA support influences the involvement of other funding partners.		X	X				Low
4.2 How and to what extent has the sub program contributed to immediate outcomes (P&G) [lxxi]: Increased expansions and modernizations Increased technology acquisition Increased productivity and business skills Increased domestic marketing Increased capacity for establishment and growth Increased productivity and business skills/capacity	PMF targets have been established and achieved (see Appendix X) There is sufficient evidence to argue the contribution of the sub program activities to the achievement of immediate outcomes . Clients attribute the achievement of immediate outcomes to ACOA’s P&G programming. Increased expansions (production capability) Increased modernizations (improvement of processes / products) Increased technology acquisition (implementation of new technology) Enhanced productivity and business skills capacity Enhanced marketing capabilities and activities Enhanced knowledge / skills contributing to the ability to start or grow a business Business creation The achievement of immediate outcomes is similar or greater to that observed in the previous evaluation (accounting for changes in context and programming). Note the comparison will only be made where similar data from previous evaluations exists.		X	X			X	Med-High
4.3 How and to what extent has the sub program contributed to intermediate and long-term outcomes (P&G): Increased sales and productivity Increase economic benefits to Atlantic Canada from P&G. Improved growth and competitiveness of Atlantic SMEs	Evidence exists to support the contribution of the sub program activities to the achievement of intermediate/long-term outcomes according to expected timelines . The achievement of intermediate/long-term outcomes is similar or greater to that observed in the previous evaluation (accounting for changes in context, processes and procedures). Clients attribute, to at least a moderate extent, the achievement of intermediate/long-term outcomes to ACOA’s P&G programming. Intermediate / long-term outcomes include: Reduced costs of production Development / reach of new markets Increased market shares Increased revenues, sales volumes, profitability Improved operational efficiencies Waste reduction Jobs created or maintained Business survival Performance of ACOA supported businesses compared with unassisted businesses with respect to business survival, sales growth and labour productivity.	X	X	X			X	Med-High
4.4 To what extent has a client-centric approach to program delivery enabled client firms to access a range of supports under P&G as well as other sub-programs, as needed? Has this approach contributed to their ability to achieve productivity and growth outcomes?	Evidence that ACOA’s approach to program delivery enables SMEs to access a range of services across ED programming as required. There is evidence to suggest that this approach enhances client’s ability to achieve productivity and growth related outcomes. Note that documents such as case studies and interview data produced as part of the IBD and IC evaluations will be reviewed to answer this evaluation question.	X	X (AM, M)	X				Med-High
4.5 What, if any, unintended outcomes were achieved through the P&G sub-program?	Not applicable		X (AM, M)	X				Low
4.6 What are the facilitators/best practices and barriers/lessons learned related to delivering the P&G sub-program and achieving expected outcomes? To what extent are barriers being mitigated?	Evidence of factors facilitating the achievement of P&G expected outcomes (internal and external) are known, shared and implemented where appropriate. Evidence of factors (internal and/or external) that are impeding success of the sub program are known, and mitigation strategies are implemented where appropriate (based on level of risk and control). There is evidence that best practices and lessons learned are identified, considered, and/or implemented, and that mitigation measures are put in place. Evidence that ACOA takes into consideration the findings/recommendations of previous evaluation /audits/reviews in improving delivery and considering alternative approaches related to P&G.	X	X (AM, M, E/S)	X				Med
Issue 5: Demonstration of Efficiency and Economy
5.1 To what extent are delivery costs reasonable in relation to outcomes achieved?	Delivery costs across regions have remained stable or decreased since last evaluation (accounting for changes in context, processes and procedures), and reasonably compare to one another within the context of factors affecting regional delivery. ACOA cost of delivering $1 G&Cs is reasonable compared to the previous evaluation and to delivery of similar programming. Costs compare reasonably to other similar service delivery models (if applicable)	X			X	X		Low
5.2 In the context of the results being achieved, how and to what extent are the resources allocated to the sub-program efficiently and economically utilized?	ACOA has in place resource optimization mechanisms to ensure that the most efficient and economical means are being used to administer the programming. There is evidence that: governance structures and practices allow for efficient and economical delivery including coordination of related aspects ED, CD and PAC activities; there is a rationale/strategy linked to expected results guiding HO and regional office P&G delivery; there is effective coordination and collaboration between regions and HO as well as among regions, and this leads to: the selection of the best projects, shared intelligence and analysis, and, strategic involvement in events and activities; barriers to efficiency and economy are known and are mitigated; mechanisms are in place that allow for efficient allocation and re-allocation of resources; there is ongoing collection and use of performance measurement data in program monitoring, management and decision-making; senior management are satisfied with mechanisms for efficient and economical administration of programming;	X	X (AM, M)					Med
5.3 To what extent has program management considered and implemented alternative modes of delivery for P&G sub-program?	Evidence that program management has considered and continues to explore alternative modes of delivery within and outside the organization, and perceptions regarding possible alternative approaches to the achievement of intended P&G outcomes.	X	X (M)					Low

Appendix C – P&G Methods

The following methods were used for the P&G evaluation:

Document and Literature Review

The evaluation conducted a focused document and literature review to fill knowledge gaps and update relevant information to help address the evaluation questions, including internal (program-related) and external (published research) documentation. To calibrate the approach to the document/literature review, the evaluation team included an assessment of deliverables from the recent IC and IBD evaluations, particularly literature establishing program relevance, and case studies and survey results where the clients were also recipients of P&G funding. The evaluation team was also able to build upon the document review efforts undertaken in the elaboration of the P&G impact conceptual framework.

Program Data Review

The P&G data review consisted of gathering and analyzing data regarding program delivery, expenditures and results reporting: project, performance and operational.

Project Data: The evaluation team reviewed administrative data extracted from ACOA’s QAccess project information system which included a total of 2,727 approved projects coded to the P&G sub-program between 2008-2009 and 2013-2014. In addition to Business Development Program (BDP)-funded projects, the approved P&G projects also include projects funded by the Community Adjustment Fund (CAF) in 2009-2010 and 2010-2011, by the Atlantic Investment Partnership (AIP) in 2008-2009, BDP-AIP in 2009-2010, and by the BDP-Business Development Program grants from 2009-2010 to 2013-2014.

The analysis of project data included two important steps: the review and validation of existing project coding, and where necessary, the development of new project codes, to allow for a better understanding of the nature of some of the projects and facilitate answering some of the questions contained in the evaluation framework. For example, all BDP business support projects descriptions were analyzed to obtain a sense of the groups targeted (e.g. women, youth, industry sectors, aboriginal groups, new Canadians, etc.) No projects were incorrectly coded to P&G, and therefore none were removed. Project analysis was conducted by project type and region to profile the types of activities undertaken under P&G.
Performance Data: The P&G evaluation used existing performance data collected as part of ACOA’s performance measurement framework (PMF) as well as other strategies, to the extent possible. Performance measurement is the ongoing monitoring and assessment of program activities, outputs and outcomes in order to support results-based management and decision-making, as well as external reporting requirements. In addition, performance measurement is expected to result in credible and reliable data that effectively supports evaluation.

The most recently approved P&G sub-program data collection strategy includes indicators, targets and data sources for tracking outputs and expected results. The strategy allows for performance to be monitored and measured on an annual basis. Data on outputs and expected results of the programming was available for years 2009-2010, 2010-2011, 2011-2012, and 2012-2013. For the year 2008-2009 and 2013-2014, only data on higher-level expected results was available. Note that the P&G evaluation considered the reported outcomes achieved for the ED program, for which P&G programming contributed (e.g. business survival rates, sales growth and labour productivity growth of ACOA supported firms compared to unassisted firms).
Operational data: The evaluation included analysis of financial data including program expenditures and full-time equivalent (FTE) data from ACOA’s financial system. To address concerns associated with the availability and accuracy of operational data and to support the required analysis of efficiency and economy of the P&G sub-program, program staff was asked to validate estimates of resource utilization as part of the evaluation’s regional profile documentation. Corporate Finance is in the process of developing a new costing model that will enable a more streamlined approach to the analysis of operational data.

Key Informant Interviews

Interviews were conducted strategically to address knowledge gaps and to validate understanding of the programming. A total of 54 key informants (KIs) were interviewed including a cross-section of internal ED management and staff as well as managers and staff from other ACOA program areas (~40 – note that many of these were group interviews). In addition, external subject matter experts and stakeholders were interviewed (~14).

Client Survey

An internet-based survey (with telephone follow-up) was conducted with the sub-program’s direct funding recipients to collect information on the clients, activities and outcomes of P&G-related programming. To report on the integration of programming across Enterprise Development, the evaluation sought to use survey findings from the other recent and on-going evaluations, specifically for clients who had benefited from P&G assistance as well as Innovation and Commercialization or International Business Development supports.

Survey response: The overall response rate was 33% - 31% Commercial; 48% Non-Commercial. The overall sample achieved target value to allow for a margin of error of +/- 5%, with a confidence interval of 95%. Taken separately, that margin of error is 5.3% for commercial firms, and 12.6% for non-commercial (the sample size required for 5% was described as unattainable by industry standards). While we have more confidence that findings for commercial clients are more representative of the population, we included findings for non-commercial as they provide an indication of results for non-commercial projects.

Other Analyses

The evaluation team used the findings of a comparative analysis between ACOA supported firms and unsupported firms conducted by Statistics Canada in the spring of 2015. This comparison provided an indication of the impact of ACOA programming on assisted firms.

Appendix D – P&G Limitations and Mitigating Strategies

Limitations	Mitigation Strategy
Short timelines for conducting the evaluation	Use of existing/new data as required to reduce overlap of effort. Extensive consultation at outset to allow determination of key evaluation issues and to maximize usefulness. Restricted resource-intensive coding exercise to target evaluation questions.
Challenges of conducting contribution analysis: results can be influenced by multitude of factors	Limitations associated with establishing the contribution of outcomes to P&G programming were addressed by considering the proportion of ACOA assistance contributed, and relying on the client survey to assess incrementality of ACOA support (i.e. what would have happened without ACOA funds), as well as the impact the project had on outcomes achieved. Clients surveyed were also asked to reflect the extent to which outcomes resulted from the project in question.
QAccess coding errors, inconsistencies and/or missing project data	Coding verified for high-level categories; more detailed analysis re: business support, regional validation of coding. Acknowledgement of the inter-relatedness of the ED sub-programs.
Lack of information on project outcomes: performance data	Compensated for the lack of performance data with client survey to the extent possible.
Selection bias: people who participate may be different from those who decline/are not invited to participate	Invited a range of stakeholders for interviews. Invited all unique clients to complete the survey. The HOE (head of evaluation) sent notifications to potential informants explaining purpose of evaluation and encouraging participation.
Response bias: client desire to overstate positive outcomes	Clearly communicated to clients that the responses were anonymous to the Agency, and encouraged honest responses. Provided a web link for responding to the survey, demonstrating its independence from any identifying information.
Challenges associated with establishing a comparison group	The evaluation team made use of an on-going ACOA contract with Statistics Canada to compare aggregate performance data on the businesses served by ACOA with the performance data of all other businesses in the same sectors. More sophisticated comparative methods exist to further isolate the impact of programming, such as matching clients with non-clients from the outset on multiple variables such as age, employment, assets, profit margin, revenues, (etc.). This enables a comparison of changes in both groups over time, and the ability to conduct regression analysis to test hypotheses on the impact of the programming. For practical reasons, ACOA was unable to employ those methods for this study, but is exploring the possibility of using such analyses in the future.

Appendix E – Program Alignment Architecture

Strategic Outcome	Programs	Sub-Programs
A competitive Atlantic Canadian economy (1.0)	Enterprise Development (1.1)	Innovation and Commercialization (1.1.1)
		Productivity and Growth (1.1.2)
		International Business Development (1.1.3)
	Community Development (1.2)	Community Mobilization (1.2.1)
		Community-based Business Development (1.2.2)
		Community Investment (1.2.3) Infrastructure Programming (1.2.4)
	Policy, Advocacy and Coordination (1.3)	Policy (1.3.1)
		Advocacy (1.3.2)
		Coordination (1.3.3)
	Internal Services (1.4)	Governance and Management Support (1.4.1)
		Resource Management Services (1.4.2)
		Asset Management Services (1.4.3)

Source: ACOA Program Alignment Architecture, Effective April 2013.

Appendix F – P&G Logic Model

Reach	*Clients* Potential and existing entrepreneurs SMEs Business Service Organizations NGOs Educational institutions *Stakeholders/Co-deliverers* Other Government Departments Business support organizations Trainers/Consultants Atlantic Canadians Media	*Activities and Outputs*	Financing of productivity and growth projects SME development activities Business information products and services
		*Immediate Outcomes*	Increased expansion/modernization of SMEs Increased technology acquisition by SMEs Increased productivity and business skills initiatives by SMEs Increased domestic marketing projects by SMEs Increased developmental activities in support of SME establishment and growth Increased developmental activities in support of productivity and business skills Improved opportunities to acquire information needed to start and grow a business
		*Intermediate Outcomes*	Increased SME capacity for growth Enhanced SME productivity and business skills Enhanced access to information, resources and skills needed to start and grow a business
		*Productivity and growth*	Enhance Atlantic Canadian SMEs productivity and growth capacity
		*Enterprise Development*	Improved growth and competitiveness of Atlantic SMEs
		Strategic Outcome	A competitive Atlantic Canadian economy

Appendix G – P&G Conceptual/Analytical Framework

(D)

Source: ACOA, Impact Conceptual and Analytical Framework, September 2012

Appendix H – Agency P&G Business Support Details, 2008-09 to 2013-14

Business Support Projects by Type	# of projects	% of Projects	ACOA Assistance ($)	ACOA Assistance (%)
Business Support	615	100%	$72,201,062	100%
BSD	553	89.9%	$64,828,851	89.8%
Aboriginal Women	4	0.65%	$272,451	0.38%
Aboriginal Youth	4	0.65%	$236,731	0.33%
General SMEs	67	10.9%	$9,631,420	13.3%
Industry Sector	126	20.5%	$17,034,926	23.6%
New Canadians	6	1.0%	$343,824	0.5%
Seniors	2	0.3%	$34,905	0.0%
UBDC	44	7.1%	$6,657,570	9.2%
Women in Business Development	58	9.4%	$13,025,858	18.0%
Youth	213	34.6%	$15,505,917	21.5%
Youth/Seniors	1	0.2%	$17,808	0.0%
New Canadian Women	5	0.81%	$257,965	0.36%
Aboriginal BSD	5	0.81%	$207,872	0.29%
Francophone SMEs	9	1.5%	$609,592	0.8%
Infrastructure for Post-secondary	3	0.5%	$697,350	1.0%
Francophone Youth	6	1.0%	$294,662	0.4%
Study/Plan	16	2.6%	$962,813	1.3%
Modular Fabrication Facility	1	0.16%	$187,500	0.26%
Transportation Infrastructure	1	0.2%	$100,000	0.1%
Streetscape	1	0.2%	$30,000	0.0%
Partnership Development	1	0.2%	$199,000	0.3%
Post-Secondary Institution	3	0.5%	$97,084	0.1%
Performing Arts Centre	2	0.3%	$32,000	0.0%
Aboriginal Commercial Venture	3	0.5%	$209,429	0.3%
Bioscience	1	0.2%	$11,700	0.0%
Entrepreneurship	1	0.17%	$9,450	0.01%
Tourism Operation	1	0.16%	$56,250	0.08%
Challenges access capital by SMEs	1	0.16%	$30,400	0.04%
Technical Training	11	1.8%	$1,685,666	2.3%
Industry Sector	11	1.8%	$1,685,666	2.3%
Tourism	12	1.9%	$1,735,296	2.4%
Training Centre	1	0.16%	$429,600	0.60%
Event	7	1.1%	$340,696	0.5%
Tourism Operation	4	0.65%	$965,000	1.34%
Industry Sector Marketing	23	3.71%	$2,988,436	4.14%
Industry Sector	21	3.39%	$2,908,561	4.03%
Francophone SMEs	2	0.32%	$79,875	0.11%
Grand Total	615	100%	$72,201,062	100%

Appendix I - Organizations Providing Funding to Atlantic Canadian SMEs

The landscape of all of the different programs offered in the region by various funders is complex. However, there do not appear to be large areas of overlap. The various funding agencies tend to fund different parts of the financing continuum.

For example, in terms of other federal funding entities, the National Research Council (NRC) focuses on R&D, technology and innovation. The Business Development Corporation (BDC) does offer loans to SMEs, and is sometimes a co-funder on projects with ACOA. However, BDC loans differ from ACOA’s contributions in that they charge interest and require funding to be secured. While the Export Development Corporation’s (EDC’s) focus is primarily on international export development, in 2011 its mandate was expanded and now includes support for domestic trade capacity building. Key informants perceived EDC as working in other areas than ACOA, and no survey respondents mentioned EDC as a co-funder.

The Community Business Development Corporations (CBDCs) fund higher-risk contributions for smaller amounts, charge interest, and support sectors not supported by ACOA (e.g. retail).

More detail about the characteristics of these funding organizations can be found in the chart below.

Type of Financing Organization	Characteristics
ACOA P&G Funding	Rural and urban coverage High risk tolerance; participation by other lenders often contingent on ACOA participation Non-secured, interest-free contributions Flexible repayment terms and periods Some sector restrictions – retail/wholesale, real estate, government services, personal services and primary production excluded Project selection criteria include consideration of competitive effects and scope to achieve incremental economic benefit
Community Business Development Corporations (CBDCs)	Based in rural communities and community-focused High risk tolerance, often lender of last resort Requires security, and charges commercial rates of interest Wide range of sectors covered, including retail Generally smaller contributions, often oriented toward start-ups Aim is to support clients, often to the point where they are “bankable” and/or their financing needs may be met by ACOA or other providers Project selection criteria include consideration of competitive effects
Provincial ministries and agencies	Targeted uses, sectors of focus and structuring of financing vary, depending on provincial priorities and availability of funding Generally require security Generally include working capital loans Often positioned similarly to ACOA and enable more leveraging on ACOA funding
Credit unions and caisses populaires	Limited amount of lending activity for SMEs; greater emphasis on consumer services Require security, and lend on commercial terms Community-based and community-oriented Level of involvement with rural SMEs varies across the Atlantic provinces
Business Development Bank of Canada	Positioned between ACOA and chartered banks: higher risk tolerance and flexibility than chartered banks, but on commercial lending terms Provide term loans (including working capital loans), subordinate financing, and guarantees for operating lines of credit Provide both security and asset-backed financing Not constrained by incrementality and competitiveness conditions: focus is on being a “complementary lender” Broader sector coverage, including retail, wholesale, real estate Active in both urban and rural areas
Chartered banks	Unwilling to accept same level of risk as the CBDCs, ACOA and provincial funding agencies Require security, with a preference for tangible assets Limited presence in rural areas Decision-making removed from local level Prefer to see ACOA participating (as do other providers) as a means of spreading risk without having to reduce security

Footnotes

[i] Accountability requirements are outlined in the Treasury Board’s Policy on Evaluation as well as the Directive on the Evaluation Function and the Standard on Evaluation for the Government of Canada. For small agencies, all grant and contribution spending must be evaluated every five years.

[ii] According to the Treasury Board Directive on the Evaluation Function, demonstration of efficiency and economy is an “assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.”

[iii] Only the Productivity and Business Skills Initiative component of the Innovation Evaluation is relevant to P&G.

[iv] http://www.acoa-apeca.gc.ca/eng/Accountability/AuditsAndEvaluations/Pages/EvaluationoftheAtlanticCanadaOpportunitiesAgencyFinancingContinuumProgramSubactivityFinalReport.aspx

[v] http://www.acoa-apeca.gc.ca/eng/Accountability/AuditsAndEvaluations/Pages/EvaluationoftheAtlanticCanadaOpportunitiesAgency’sEntrepreneurshipandBusinessSkillsDevelopmentProgramSub-activity.aspx

[vi] P&G Terms of Reference, approved by ACOA’s President on February 4, 2015.

[vii] Accountability requirements of the Financial Administration Act (FAA), are outlined in the Treasury Board Policy on Evaluation as well as the Directive on the Evaluation Function and the Standard on Evaluation for the Government of Canada. For small agencies, all grant and contribution spending must be evaluated every five years.

[viii] A horizontal evaluation of the Canada Business Network led by Industry Canada was completed in June, 2014. The evaluation identified positive impacts in terms of relevance and performance. The evaluation resulted in two recommendations. One is aimed at improving design and delivery through consideration of international best practices and client expectations, while re-examining both the governance and service delivery models. The second targets the establishment of standardized performance measures and financial tracking in order to enhance the consistency of performance reporting across the network. http://www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03696.html.

[ix] Funding for these projects are included in total expenditures as they represent programming delivered by ACOA P&G staff in the evaluation’s assessment of programming efficiency and economy.

[x] The ED planning study was completed in 2012 to clarify the objectives, scope and utilization of the three ED sub-program evaluations. It helped gauge evaluation readiness by testing the program logic/theory, building program profiles, clarifying data requirements, and testing data availability and quality prior to the evaluation being conducted.

[xi] While YEDI and WBI have ended, the BDP continues to support projects related to youth and women.

[xii] PBSI was originally an initiative funded under the former Innovation sub-activity. These activities became a project type under the P&G sub-program when it was established in 2011-2012.

[xiii] The model for sub-delegating authority varies slightly by region. The delegated authority and management structure in each region were validated as part of the evaluation process.

[xiv] Totals for “ACOA Assistance” are presented based on project approval date, meaning the total assistance is reported in the year the project was approved versus the year in which the funds were disbursed. Funding approved in one year may have also been disbursed in subsequent years.

[xv] In 2009-2010 and 2010-2011, ACOA provided assistance of $2.6M to 7 BDP-CAF projects in NS and NB. While EAP projects such as those funded through CAF are not included in the scope of this evaluation in terms of evaluating their outcomes, they are represented in financial tables to provide a full picture of funding delivered by the sub-program. ACOA issued a final report to TBS on the results of the CAF initiative in January 2013.

[xvi] Timing differences related to the recognition of expenditures between QAccess and ACOA’s financial system account for the variance between the “ACOA Assistance” in Table 2 and the total “G&C” reported in Table 3.

[xvii] Note: The variance between total ACOA assistance (Table 2) and the Total in Table 3 are largely due to timing differences between QAccess and the GX Financial System.

[xviii] Expenditures included in Table 2 not subject to this evaluation include: allocation from internal services representing $3.9M ($2.1M in salaries and $1.8M in general operating) and approximately $3.2M ($2.3M in salaries and $990,000 in operating) representing Canada Business, evaluated by Industry Canada as part of the 2013-2014 Canada Business Network Evaluation. https://www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03696.html

[xix] Statistics Canada. 2012. Age and Sex 2011 counts for both sexes for Canada provinces and territories, Age and Sex Highlight Tables. 2011 Census. Statistics Canada Catalogue no. 98-311-XWE2011002. Ottawa. Released May 29,2012

[xx] Statistics Canada, Summary Tables: Population, urban and rural, by province and territory, 2011 Census of Population, www.statcan.gc.ca

[xxi] Industry Canada. The Canadian Provinces - Special Edition: Key Small Business Statistics. September 2013, www.ic.gc.ca

[xxii] See: David Chaundy et al., Meeting the Skills Challenge: Five Key Labour Market Issues Facing Atlantic Canada, Atlantic Provinces Economic Council, October 2012.

[xxiii] Industry Canada, Survey on Financing and Growth of Small and Medium Enterprises, 2011. Industry Canada website, https://www.ic.gc.ca/eic/site/061.nsf/eng/02775.html#tab2

[xxiv] Industry Canada, 2011, Ibid.

[xxv] Gary Gorman, Dennis Hanlon and Blair Winsor, GEM Newfoundland and Labrador Report 2013, Global Entrepreneurship Monitor (GEM), p.20. www.gemconsortium.org

[xxvi] Statistics Canada, Table 383-0029 Labour productivity and related variables by business sector industry, CANSIM (Database). Accessed April 9, 2015

[xxvii] Cooper H. Langford, Peter Josty and J. Adam Holbrook. Driving Wealth Creation and Social Development in Canada: 2013 GEM Canada National Report, Global Entrepreneurship Monitor, 2013

[xxviii] Statistics Canada, Table 282-0012 – Labour Force Survey Estimates, employment by class of worker, North American Industry Classification System (NAICS) and sex, annual (persons), CANSIM (Database). Accessed March 20, 2015

[xxix] OECD, Job Creation and Local Economic Development, OECD Publishing, 2014. DOI: 110.1787/9789264215009-en

[xxx] Note that contributions to associations headquartered in an urban area but serving an entire province were counted as “rural”.

[xxxi] The recent evaluation of ACOA’s Community Mobilization and Community Investment Sub-program (2013) found that over the four-year period of 2008-09 to 2011-12, under CM and CI programming, ACOA supported 76 business skills development projects for a total of $10.4M in investments. These projects were mainly targeted to general SMEs (29 projects for $4.9M), and SMEs within certain industry sectors (43 projects for 4.8M). Each region funded some BSD projects under CD, but the majority were funded in Newfoundland and Labrador (27 projects) and New Brunswick (29 projects).

[xxxii] These figures include all programming that promoted entrepreneurship as a career option, and any projects aimed at youth, including contributions to University Business Development Centres, which serve youth but also the broader business community.

[xxxiii] This decline in number of projects may not be as pronounced as the data would indicate due to the presence of multi-year projects - though most multi-year projects were captured in the evaluation data reviewed, a few could have been approved/ renewed shortly after the period ended. Another factor that influenced the decline in the number of youth entrepreneurship projects was the closure of some organizations delivering the programming in New Brunswick.

[xxxiv] Unemployment rates are for youth aged 15-24 for the year 2014. Source: Statistics Canada, Table 282-0002 Labour force survey estimates (LFS), by sex and detailed age group, annual, CANSIM (Database). Accessed April 4, 2015.

[xxxv] Industry Canada. The Canadian Provinces - Special Edition: Key Small Business Statistics. September 2013, www.ic.gc.ca

[xxxvi] The decline in number of projects approved annually is not due to the existence of multi-year projects. Since the end of WBI, there was one multi-year project approved in 2011-2012 (ending in 2013). There were two multi-year projects approved in 2013-2014, but these would not affect any of the annual totals in the time period of the evaluation.

[xxxvii] David Chaundy et al., Meeting the Skills Challenge: Five Key Labour Market Issues Facing Atlantic Canada, Atlantic Provinces Economic Council, October 2012.

[xxxviii] Sandi Findlay-Thompson and Karen Blotnicky, Pan-Atlantic Needs Assessment of Women-Owned Growth-Oriented Businesses, Atlantic Canada Women in Export Working Group (ACWIE), December 2012, p. 5 http://www.centreforwomeninbusiness.ca/site/media/CWB/Documents/ACWIE%20-ENG(1).pdf

[xxxix] Leslie Preston, “Canadian Women Leading the Charge into Entrepreneurship”, TD Economics Special Report, January 16, 2015 http://www.td.com/document/PDF/economics/special/WomenEntrepreneurs.pdf

[xl] Benjamin Tal, “Start-Ups – Present and Future”, In-Focus, CIBC Economics, September 25, 2012

[xli] Chaundy et al., Ibid.

[xlii] Statistics Canada, Table 282-0012 – Labour Force Survey Estimates, employment by class of worker, North American Industry Classification System (NAICS) and sex, annual (persons), CANSIM (Database). Accessed March 20, 2015

[xliii] Status of Women Canada, “An Overview of GBA+ in the Federal Government”. http://www.swc-cfc.gc.ca/gba-acs/overview-apercu-en.html , Date modified: 2015-05-14

[xliv] Status of Women Canada, “GBA+ as a key competency for public servants”, http://www.swc-cfc.gc.ca/gba-acs/competency-competence-en.html, Date modified: 2013-09-30

[xlv] Atlantic Canada Opportunities Agency, 2013-14 Report on Plans and Priorities (RPP), 2014.

[xlvi] Treasury Board Secretariat, Whole-of-government framework, Government of Canada, date modified: February 23, 2015.

[xlvii] Governor General of Canada, “Seizing Canada’s Moment: Prosperity and Opportunity in an Uncertain World”, Speech from the Throne to Open the Second Session of the Forty-First Parliament of Canada, October 16, 2013.

[xlviii] Governor General of Canada, “Here for All Canadians: Stability. Prosperity. Security.” Speech from the Throne to Open the First Session of the Forty-First Parliament of Canada, June 3, 2011. http://publications.gc.ca/collections/collection_2011/gg/SO1-1-2011-eng.pdf

[xlix] Atlantic Canada Opportunities Agency Act, R.S.C. 1985, c. 41 (4th Supp.). Current to March 24, 2015. http://laws-lois.justice.gc.ca/eng/acts/A-13.7/

[l] Other regional development agencies with similar types of programming include the Federal Economic Development Agency for Southern Ontario and Canada Economic Development for Quebec Regions.

[li] The proportion of ACOA’s contribution is influenced by the nature and eligibility of activities included in the total project costs, as well as the client’s ability to contribute their own funds and to obtain funds from other sources.

[lii] These figures are not comparable to previous evaluations due to gaps in comparable data, but provide a sense of the nature of the incrementality of ACOA funding.

[liii] The analysis represented here includes only the first four years of the timeframe, to maximize accuracy given that for subsequent years, P&G projects also included projects that were previously counted under other areas of programming.

[lix] Statistics Canada. Table 551-0006 - Canadian business patterns, location counts, employment size and North American Industry Classification System (NAICS), by Canada and provinces, June 2014, semi-annual (number), CANSIM (Database). Accessed: June 19, 2015

[lv] This includes projects in the North American Industry Classification System (NAICS) categories 31-33 Manufacturing.

[lvi] NAICS category 11

[lvii] NAICS category 54

[lviii] ACOA was not able to match SMEs from the comparison group on factors affecting performance. For example, if a greater proportion of comparison group SMEs would not qualify for ACOA assistance due to lack of experience or financial stability, this could partly explain the outcomes observed for that group.

[lix] Unassisted firms are defined as businesses that have received no financial assistance from ACOA since inception.

[lx] Operational efficiency is largely concerned with how inputs are being used and converted into outputs that support the achievement of intended outcomes. http://www.tbs-sct.gc.ca/cee/pubs/ci5-qf5/ci5-qf5-eng.pdf

[lxi] This perspective on efficiency is generally concerned with the big picture of whether the resources consumed in the achievement of outcomes were reasonable in light of issues such as the degree of outcome achievement, the program’s context and the alternatives to the existing program. http://www.tbs-sct.gc.ca/cee/pubs/ci5-qf5/ci5-qf5-eng.pdf

[lxii] Higher cost per project for FC Evaluation due to much lower volume of projects (924 over 5 years vs. 2727 for P&G) combined with significantly higher “Other” operating (Avg. $5.5M per year vs $2.4M per year for P&G).

[lxiii] Budget 2010 announced that operating and salary budgets would be frozen at their 2010-2011 levels for fiscal years 2011-2012 and 2012-2013. ACOA also implemented a series of measures during those fiscal years to address reductions to the general operating costs. These included caps on travel, hospitality, conferences and professional services. Budget 2011 identified savings of $15.2 million by 2014-2015 for ACOA. A review of ACOA’s activities allowed the Agency to identify ways to consolidate various functions, reduce the duplication of work, and eliminate non-priority activities without impacting the Agency’s ability to deliver on its core programs.

[lxiv] The large proportion of Special/ Professional services in NS was due to a substantial fund for administering consulting and advisory services which did not exist for other regions (NS was an outlier here).

[lxv] The impacts may be understated to the extent that many of the projects are still on-going or have only recently been completed and impacts may yet increase.

[lxvi] Treasury Board of Canada Secretariat, Directive on Delegation of Financial Authorities for Disbursements http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=17060

[lxvii] ACOA’s Delegated Authorities chart provides regional VPs with the authority to approve projects up to $500K. Any proposals in excess of $500K are required to go through IRC approval process and obtain the President’s approval.

[lxviii] According to Black’s Law Dictionary http://thelawdictionary.org/, statute of limitation refers to: “A law that bars claims after a specified period; specif., a statute establishing a time limit for suing in a civil case, based on the date when the claim accrued (as when the injury occurred or was discovered).”

[lxix] Provisionally Repayable Contributions Audit (2012) recommended Special Accounts continue training sessions on the agreements, monitoring requirements and troubleshooting and when to seek advice from Special Accounts (re: statute of limitations).

[lxx] Coverage of judgment criteria will depend on the existence of available data and level of effort associated with the collection of additional data. Judgment criteria may be modified or removed if the level of effort outweighs values to decision making.

[lxxi] A revised P&G sub-program logic model is currently under development and changes to expected outcomes are pending. The expected outcomes that appear in the evaluation framework have been modified from the current P&G logic model at the time of TOR development based on discussion with program management.

Page details

Date modified:: 2024-04-04

Evaluation of the Atlantic Canada Opportunities Agency’s Productivity and Growth Sub-Program FINAL REPORT

Glossary of Acronyms

Table of Contents

List of Figures

List of Tables

Acknowledgements

Executive Summary

1.0 Introduction

1.1 Evaluation Overview

1.2 Evaluation Design and Methodology

1.3 Evaluation Strengths, Limitations and Mitigating Measures

2.0 Profile of the Productivity and Growth Sub-program

2.1 Program Theory

2.2 Program Accountability and Governance

2.3 Project Approval and Signing Authority

2.4 Expenditure Profile

2.5 Program Priorities and Project Profile

3.0 Findings: Relevance

3.1 Continued Need for Programming

3.2 Alignment with Government Priorities, Federal Roles and Responsibilities

4.0 Findings: Performance – Effectiveness

4.1 Incrementality

4.2 Achievement of Expected Outcomes

4.2.1 Commercial Programming

4.2.2 Non-Commercial Programming

4.3 Client-Centred Approach and Influence on Outcomes

4.4 Barriers and Facilitators to the Achievement of P&G Outcomes

5.0 Findings: Performance – Efficiency and Economy

5.1 Efficient Utilization of Resources

5.2 Mechanisms Supporting Efficiency and Economy

5.2.1 Factors Contributing to Efficiency and Economy

5.2.2 Opportunities for Improving Efficiency and Economy

6.0 Conclusions and Recommendations

6.1 Conclusions

6.2 Recommendations

Appendix A – P&G Evaluation Management Action Plan

Appendix B – P&G Evaluation Framework: Questions, Judgment Criteria and Methods

Appendix C – P&G Methods

Appendix D – P&G Limitations and Mitigating Strategies

Appendix E – Program Alignment Architecture

Appendix F – P&G Logic Model

Appendix G – P&G Conceptual/Analytical Framework

Appendix H – Agency P&G Business Support Details, 2008-09 to 2013-14

Appendix I - Organizations Providing Funding to Atlantic Canadian SMEs

Footnotes

Page details