ACOA Innovation Programming Evaluation Report 2020

Download the printer friendly version (PDF)

2013-2014 to 2018-2019

Evaluation and Advisory Services

Atlantic Canada Opportunities Agency

January 2020

Acronyms

ACOA Atlantic Canada Opportunities Agency

AIF Atlantic Innovation Fund

BAIs business accelerators and incubators

BDC Business Development Bank of Canada

BDP Business Development Program

DRF departmental results framework

EAC evaluation advisory committee

EAS Evaluation and Advisory Services

ED Enterprise Development

FAA Federal Accountability Act

FTE full-time equivalent

GAC Global Affairs Canada

GBA+ gender-based analysis plus

GC Government of Canada

G&Cs grants and contributions

IRCC Immigration, Refugees and Citizenship Canada

ISED Innovation, Science and Economic Development

MAP management action plan

NRC-IRAP National Research Council of Canada -– Industrial Research Assistance Program

O&M operations and maintenance

PAF project approval form

REGI Regional Economic Growth through Innovation

RDA regional development agency

R&D research and development

SMEs small and medium-sized enterprises

TBS Treasury Board Secretariat

Figures

Tables

Acknowledgements

This evaluation provides the Atlantic Canada Opportunities Agency's (ACOA) management with objective, neutral evidence on the relevance and performance of the Agency's Innovation programming. ACOA's Evaluation and Advisory Services unit completed this evaluation of Innovation programming with advice from an evaluation advisory committee (EAC).

I offer deep gratitude to the members of the EAC for their advice and support throughout the study. Their contributions helped to ensure the relevance and usefulness of the evaluation. Very special thanks go to the external members of the EAC: Matt MacPhee, Senior Project Manager, Saint Mary's University Entrepreneurship Centre; Beth Mason, CEO, Verschuren Centre, Cape Breton University; and, Maurice Turcot, Senior Evaluation Analyst, Canada Economic Development for Quebec Regions. Internal members of the committee include Joe Cashin, Michael Dillon, Kalie Hatt-Kilburn, Josh Jenkins, Anne-Marie Léger, Jeff Mullen, Hélène Robichaud, Daniel Scholten and Karl Tee.

The evaluation team is grateful to the external key informants as well as the many ACOA employees from across Atlantic Canada who provided their time and knowledge in support of this evaluation.

Finally, to the evaluation team members – Deanna Slattery-Doiron, Anouk Utzschneider, Nicole Saulnier, Jasmine Arsenault, Gaétanne Kerry, Paula Walters and Paul-Émile David – your dedication, professionalism and commitment to excellence is exceptional. Thanks to each of you.

Colleen Goggin

Director, Evaluation and Advisory Services

Summary

The purpose of this evaluation is to assess the relevance and performance of ACOA's Innovation programming over a six-year period, from 2013-2014 to 2018-2019. The evaluation fulfills Government of Canada (GC) accountability requirements and provides the Agency's management with systematic, neutral evidence to support continuous program improvement. The methodology included interviews with 42 key informants, a client survey with an overall response rate of 48%, two case studies, a document and literature review, and an analysis of available performance data.

ACOA's investments through Innovation programming aim to create a robust regional innovation ecosystem and accelerate the growth of small and medium-sized enterprises (SMEs) through the delivery of both financial and non-financial supports. Projects and initiatives focus on the following objectives: capacity for research and development (R&D), commercialization, productivity and business skills, and the attraction and retention of talent; development and demonstration of new or improved technologies, products, processes or services; and collaboration on innovative technology and commercialization projects.

Over the evaluation period, ACOA supported more than 850 innovation projects across Atlantic Canada through grant and contribution (G&C) expenditures and non-financial supports. The Agency expended $508 million, with 91% of this spending through G&Cs and the remainder to support operational delivery costs.

To what extent does the programming address a unique and demonstrable need for Atlantic Canadians?

The evaluation finds that there is a strong and continued need for ACOA's Innovation programming. Challenges to business growth exist to the same degree or are greater than reported in previous evaluations. In particular, the evaluation reports the following: a shortage of skills and labour influenced by demographic shifts; a need for greater awareness and use of innovative technologies; and a perceived need for greater access to specific types of capital (i.e., early-stage, growth stage and angel and venture capital). The program delivery model, with local presence as well as convening and pathfinding roles, allows the Agency to be responsive to the diverse and changing economic needs across the region. ACOA has developed or strengthened collaborations to better address immigration, skills and labour, and funding.

The Innovation programming is unique, complementary to other programs, and aligned with current federal government priorities on innovation and business growth. It is clearly aligned with ACOA's mandate to lead economic development in Atlantic Canada through support to SMEs. The programming offers unique features, including flexible, non-dilutive funding options and non-financial supports focused on promoting the growth of SMEs in the region.

To what extent is ACOA contributing to business growth in Atlantic Canada?

The evaluation finds that projects have achieved the expected outcomes of the Innovation programming. Over the evaluation period, ACOA provided $463 million to commercial and non-commercial clients to support the growth of SMEs through Innovation programming. ACOA's funding and other supports contribute to a strengthened innovation ecosystem, the commercialization of new or improved technologies, products, processes or services, and productivity improvements. Statistics Canada data reveals that ACOA clients report higher R&D and productivity compared to non-supported firms. In a survey of all ACOA Enterprise Development (ED) clients, 77% reported that projects contributed to commercialization; 75% indicated that their projects improved the efficiency or capacity of operations; and 67% said that it involved technology acquisition, adaptation or adoption, new equipment or renovations. Key informants, case studies and a document review indicated that ACOA contributes to a strengthened ecosystem, especially through funding to business accelerators and incubators (BAIs) and the important role the Agency plays in developing and maintaining collaborations, pathfinding and convening.

While there are opportunities to improve the ability to capture and report on longer-term impacts of programming, some initiatives currently under way hold promise for improving the quality and availability of performance data. These initiatives include the implementation of a new G&C information system and agreements with Revenue Canada and Statistics Canada, as well as the development of a new standardized national performance framework for BAIs.

The absence of programming would have had a negative impact on the achievement of results. Ninety-one percent of ACOA clients said the absence of ACOA funding would have had major negative repercussions on their project. Key informants and a literature review revealed that the start-up ecosystem (BAIs) would be weaker without ACOA supports.

To what extent is ACOA taking a client-centric approach to program design and implementation?

The evaluation found that the design and delivery of Innovation programming contributes to a client-centric approach to some extent. ACOA employs mechanisms to focus on client needs, including the provision of financial and non-financial supports, program flexibility and delivery model, internal coordination on priorities, partnerships, and collaborations that enable leveraging of funding. In a survey of ACOA clients, 83% of innovation-related clients reported that ACOA's non-financial supports are important.

Some opportunities exist to take a more strategic approach to program delivery and risk. Some factors impede client focus to some degree, including program uncertainty during the evaluation period (e.g., the transition to new innovation programming); ACOA's risk tolerance; aspects of internal and external coordination; and broad economic challenges (e.g., demographics, capital and business culture). A national review of federal innovation programming led to the implementation of new programming (i.e., the Regional Economic Growth through Innovation [REGI] program). Early implementation of REGI has been positive with no major impacts on the ability to fund projects and improved internal clarity on eligibility. However, key informants suggest gaps in the spectrum of federal funding may exist and are influenced by the unique needs in Atlantic Canada. Key informants suggested that coordination with federal partners may decrease the risks of funding gaps.

ACOA contributes to increased collaboration among stakeholders and clients. The Agency's expanded and more deliberate role in convening and pathfinding has developed new partnerships to address changing priorities. In particular, ACOA support enables projects to leverage substantial funds from other organizations at a level consistent with previous evaluations. ACOA also funds certain non-commercial organizations (e.g., Springboard, BioNova, BAIs) that encourage collaboration and business innovation. A document review and key informants indicated that recent efforts to improve coordination among BAIs are positive and must continue. However, key informants also indicated that ACOA should take a more coordinated and consistent approach to funding BAIs to improve their ability to pivot strategically, collaborate and avoid duplication of efforts.

Innovation programming contributes to inclusive growth to some extent. Inclusive growth includes promoting the participation of diverse groups and rural communities to support economic development in the region. The Agency made some efforts toward inclusive growth through funding and collaborations (e.g., immigrants, Indigenous people and women). However, there is growing internal recognition of the need to better support under-represented groups to facilitate economic growth, address skills and labour gaps, and better align with government priorities. Key informants also indicated that ACOA's 28 points of service across the region position the Agency to meet the unique needs of rural businesses.

The evaluation found that ACOA delivers Innovation programming in a cost-effective manner. Internal costs were proportionate to the amount of funding delivered. There was a high degree of client satisfaction with ACOA's service features. Recent internal operational changes are favouring more efficient program delivery (e.g., project approval form [PAF], streamlined approval process, and planned client information management system). Key informants and a review of data identified questions related to efficiency, including project approval times, performance data and ACOA's risk tolerance levels.

Recommendations

Based on the findings and conclusions, the evaluation makes three recommendations for senior management's consideration.

Recommendation 1: Build upon current Agency efforts to address any gaps in funding that may impact the scale up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners.

Recommendation 2: Examine AIF funding in relation to the current needs of SMEs and the priorities of the GC to confirm the strategic direction for the program and any changes to governance and funding processes.

Recommendation 3: Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes and better coordination among stakeholders while recognizing unique regional and SME needs.

Recommendation 4: Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration.

1.0 Introduction

This report presents the results of the evaluation of ACOA's Innovation programming. The programming supports innovation among SMEs in Atlantic Canada through direct funding and non-financial supports. The evaluation builds on the recent national review of innovation programming conducted by the Treasury Board Secretariat (TBS) to support decision making and the monitoring of this critical element of ACOA's portfolio.

ACOA is one of six federal regional development agencies (RDAs) in Canada. The Agency has a mandate to promote economic growth in Atlantic Canada by helping SMEs become more competitive, innovative and productive. It works with diverse communities to develop and diversify local economies. ACOA’s funding programs are delivered by its skilled workforce, which provides advice, convenes stakeholders and leverages funding for clients across the region.

Atlantic Canada, composed of four provinces (New Brunswick, Prince Edward Island, Nova Scotia and Newfoundland and Labrador) and over two million residents, has undergone massive change in recent years, becoming more innovative and championing a growing start-up ecosystem. In fact, Atlantic Canada showed the greatest change in ability to innovate when compared with the rest of the country, increasing from 45.8% in 2010-2012 to 69.1% in 2015-2017.Footnote (1) Startup Genome identified Atlantic Canada as one of the top 14 ecosystems in the world for "bang for buck," number one in early-stage funding for activation ecosystem globally, and fourth for high-growth activation ecosystem globally. Footnote (2) Entrevestor reports that from 2011 to 2017, Atlantic Canada start-ups produced 27 exits, including Radian6 ($326 million) and Q1Labs (reportedly more than $600 million).Footnote (3)

The evaluation covers a six-year period, from April 1, 2013 to March 31, 2019. It assesses the relevance, effectiveness and efficiency of the design and delivery of Innovation programming under the Research and Development, and Commercialization and the Innovation Ecosystem program pillars of ACOA's departmental results framework (DRF).

This report is organized into five main sections:

  • ACOA's Innovation programming
  • Evaluation approach
  • Findings
  • Conclusions
  • Recommendations

WHY EVALUATE PROGRAMS?

ACOA conducts evaluations of funding programs every five years to meet accountability requirements as well as to support senior management information needs. Evaluations support accountability, innovation and evidence-based decision making within the GC by:

  • examining the relevance and performance of programs, initiatives and policies; and
  • identifying improvements through management action plans.

ACOA's Innovation programming was last evaluated through the Evaluation of ACOA's Innovation and Commercialization Sub-program (2015).

For more information, see the Treasury Board Secretariat's Policy on Results and the Federal Accountability Act.

Appendices contain additional information: a one-page summary of findings, conclusions and recommendations (Appendix B); the management action plan (MAP) (Appendix C); the evaluation analytical framework (Appendix D); and the evaluation terms of reference (Appendix E).

2.0 Innovation programming

ACOA recognizes the importance of innovation for business growth. Its Innovation programming is designed to help grow and scale SMEs through the delivery of funding and other supports to increase:

A logic model that describes the links between program activities, outputs and expected outcomes can be found in Appendix A.

2.1 Delivery approach

ACOA's Innovation programming combines funding with non-financial supports such as advice, help with convening partners and leveraging other sources of funding. The programming is sustained financially by three G&C programs: the Atlantic Innovation Fund (AIF), the Business Development Program (BDP) and the REGI program.

The primary clients for Innovation programming are SMEs in Atlantic Canada. In addition, non-commercial clients include industry associations, BAIs, educational and research institutions, and other non-profit organizations.

The program is delivered through a continuous intake model, meaning that proponents submit project applications to ACOA at any point throughout the year. Agency program officers in regional offices across Atlantic Canada work with clients to develop projects, including identifying and convening partners, leveraging other sources of funding, and providing advice and guidance. They evaluate applications for G&Cs and manage approved projects through to the delivery of results. Table 1 provides details on the priorities and processes of each G&C program.

Table 1: Overview of ACOA programming
  Business Development Program (BDP) Atlantic Innovation Fund (AIF) Regional Economic Growth through Innovation (REGI)
Focus Start up, expand or modernize businesses; as well as not-for-profit organizations that provide support to Atlantic businesses. Develop and bring to market innovative, new or improved products or services. Two streams:
  1. Accelerate business growth and enhance productivity and competitiveness.
  2. Create and grow inclusive regional ecosystems that support innovation and business growth.
Notes Launched in 1995; projects align with ACOA’s priorities and provide economic benefits. Launched in 2001; support for R&D, with strong commercialization potential; regional project review board; continuous intake since 2014. Launched in October 2018 following TBS’s national review of innovation; two streams + Women Entrepreneurship Strategy.

2.2 Program alignment and financial resources

All federal departments must have a DRF that outlines their core responsibilities, expected results and programs. Innovation programming is part of ACOA's DRF under the "technologies" pillar, with the expected result that "businesses invest in the development and commercialization of innovation technologies in Atlantic Canada." For more information on the Agency's DRF, the program inventory and performance indicators see ACOA's DRF and program inventory of record as presented in the Agency's 2018-19 Departmental Plan.

Innovation programming expenditures include the three G&C funding programs (e.g., AIF, BDP and REGI) as well as operations and maintenance (O&M). As shown in Table 2, between April 1, 2013 and March 31, 2019, the Agency invested almost $463 million of G&C funding in 850 projects. The Agency also expended a further $45 million of O&M on salaries (93%) and on travel and other operating expenses (7%). In 2018-2019, there were approximately 70 full-time equivalent employees at ACOA directly delivering Innovation programming.

Table 2: Annual Innovation programming expenditures by fiscal year, 2013-2014 to 2018-2019
Fiscal year G&C ($M) O&M ($M) Total ($M)
Salaries Other
2013-2014 $ 75.5 $ 7.1 $0.7 $ 83.3
2014-2015 $ 72.2 $ 7.5 $0.4 $ 80.1
2015-2016 $ 69.3 $ 6.7 $0.3 $ 76.3
2016-2017 $ 64.4 $ 6.5 $0.6 $ 71.5
2017-2018 $ 82.5 $ 7.1 $0.6 $ 90.2
2018-2019 $ 98.6 $ 6.8 $0.5 $105.9
Total $462.5 $41.6 $3.1 $507.6

Source: ACOA GX financial database (extracted May 10, 2019).

2.3 Program implementation context

A number of contextual changes affected the delivery of the Innovation programming over time. Table 3 summarizes these changes and their impacts on the programming over the evaluation period. Most important, innovation became a greater priority of the federal government, with a national review of innovation programming across departments and the implementation of the new REGI program.

Table 3: Implementation context
Context Implementation context Impacts on the program
Federal Changes in government priorities in 2015 included greater emphasis on innovation, start-ups, advanced manufacturing, the ocean supercluster, international trade and investment, and supporting diversity and inclusiveness. In 2017-2018, TBS undertook a horizontal review of innovation, which explored 92 program streams across 20 federal organizations.
  • Implementation of the REGI program across all RDAs.
  • Innovation funding programs focus on: business scale up and productivity, and regional innovation ecosystems.
  • Need to navigate new and existing programming.
Provincial Shifting provincial government priorities, approaches and availability of financial resources to support regional economic development.
  • Importance of establishing new funding arrangements and partners, often with less direct financial aid for SMEs from provincial governments.
Innovation ecosystem The innovation ecosystem in Atlantic Canada is made up of a complex and shifting range of actors, including four provincial governments, federal departments, universities and research institutes and a growing number of BAIs and other business support organizations.
  • Need for ACOA to work with a vast number of new and existing partners and to contribute to the development of an innovation ecosystem that champions the growth and competitiveness of SMEs.
Departmental Internal governance changes have included the launch of pan-Atlantic priority files to better address government priorities.
  • Larger draw on staff capacity to align projects with these priority files in a matrix model, and increased complexity in reporting and governance structures.

3.0 Evaluation approach

This section outlines the approach for evaluating the Innovation programming, including methods, governance, and methodological strengths and limitations.

3.1 Evaluation questions

Based on the planning consultations and requirements of the Policy on Results and the Federal Accountability Act, the evaluation team identified three evaluation questions to guide the collection of data and the analysis of results. The questions reflected the need to examine the relevance and performance of the programming while focusing on better understanding the current needs of clients, early impacts from the implementation of the REGI program, and contributions to start-ups and the ecosystem.

Table 4 outlines the strategic questions with sub-questions.

WHAT IS INNOVATIVE ABOUT THIS EVALUATION?

This report is Part 1 of an integrated, two-part approach to evaluating the Agency’s Enterprise Development (ED) programming. A second report, focused on the Agency’s Growth and Trade programming, will be completed in May 2020. This integrated approach allows for efficiencies through shared planning and data collection processes as well as one EAC.

Table 4: Evaluation Questions
1. To what extent is the programming addressing a unique and demonstrable need for Atlantic Canadians?
     a. What are the economic development needs in Atlantic Canada? To what extent does the programming respond to these economic development needs?
     b. To what extent is the programming unique and complementary to other programming offered in the region?
2. To what extent is ACOA contributing to business growth in Atlantic Canada?
     a. How has ACOA contributed to key expected longer-term outcomes: a strengthened innovation ecosystem, increased commercialization, increased SME productivity, and increased exports and investment?
     b. What factors facilitate or impede the achievement of outcomes?
     c. How does the program align with current federal and ACOA strategic priorities? To what extent do investments in newer priorities contribute to business growth in Atlantic Canada?
     d. What impact would the absence of program funding have on Enterprise Development (i.e., incrementality) in Atlantic Canada?
3. To what extent is ACOA taking a client-centric approach to program design and implementation?
     a. To what extent have the design and delivery of the program addressed the business growth needs of clients in the ecosystem?
     b. To what extent has ACOA contributed to increased collaboration (e.g., convening, pathfinding and leveraging) among stakeholders and clients? Best practices? Lessons learned?
     c. To what extent and how does the program design and implementation contribute to inclusive growth (e.g., language minorities, youth/seniors, Indigenous people, new immigrants, gender, rural/urban, etc.)?
     d. To what extent have recent internal operational changes resulted in more efficient program delivery (e.g., program simplification, Agency-wide teams for newer priorities, capacity of account managers and integration of GBA+)?

3.2 Data collection methods

The evaluation used both qualitative and quantitative data collection methods (Table 5) to address the evaluation questions. The choice of methods was determined based on their relevance and reliability, data availability and costs.

Table 5: Summary of data collection methods
Data collection tools and objectives Sources
Document and literature review
Validate economic development needs, alignment with government priorities and broad outcomes
  • 80 internal documents, including policy reports and ACOA performance measurement reports
  • 35 external documents, including academic literature, Statistics Canada reports and other strategic reports
  • Speeches from the Throne, federal budgets, other strategic priority documents
Internal data
Document program design, implementation, nature of project, client type and performance
  • ACOA financial data (GX system)
  • ACOA performance data (QAccess system)
Key informant interviews
Assess relevance and performance from the perspective of various stakeholders
  • 42 interviewees: external stakeholders and internal stakeholders representing Programs and Policy units
Case studies
Illustrate factors that lead or contribute to program impacts
  • Two case studies: one pan-Atlantic case study on BAIs and one on BioNova, a non-commercial client that supports health and life sciences companies to start up and scale up
Client survey
Assess relevance and performance from clients’ perspective
  • Online survey of funded ED clients with a 48% (n=774/1,618) response rate
  • This report primarily uses survey results from those ED clients who indicated their project was innovation-related because it had an applied research, technology development or commercialization aspect (n=619)

3.3 Evaluation strengths and limitations

The evaluation was designed to have a calibrated approach, using mixed methods to identify strong findings while meeting TBS timelines and providing useful information to meet senior management needs. The evaluation has a number of strengths, including a skilled internal evaluation team and high stakeholder engagement through an EAC (see Appendix F) and senior management consultations. The evaluation includes detailed case studies to examine needs and longer-term impacts. The online client survey received a high response rate (48%). The evaluation considered GBA+ in its design and implementation of data collection methods, as well as in the synthesis of findings.

These practices helped to mitigate the common limitations that occur as part of most program evaluations. As outlined in Table 6, the evaluation team considered the limitations of the study and implemented a number of mitigation strategies.

Table 6: Evaluation limitations and mitigation strategies
Limitation Mitigation strategies
Programming changes
The review of innovation programming and the implementation of a new program (e.g., REGI) during the scope of the evaluation
  • Used internal analysis and consultations to frame and support results
  • Reflected changing context in the analysis of findings, conclusions and recommendations
  • Focused on early implementation and outcomes of new programming
Performance measurement data
Direct outcomes of the programming are limited
  • Collected and analyzed multiple sources of information
  • Included targeted performance questions in the client survey and key informant interviews
Timelines for evaluation
Delays due to internal programming changes and 2019 federal election
  • Calibrated design to leverage existing data and focus on key questions and information gaps
Selection bias
Clients who participated in the survey may be different from those who did not, or whose projects were not approved
  • Used other lines of evidence to support findings, such as performance data
  • Conducted interviews with a range of stakeholders, including clients
Longer-term outcomes
Program generates longer-term outcomes that are difficult to measure
  • Collected and analyzed multiple sources of information including economic data to support results.
  • Used a theory-based, realist approach to the evaluation that included engagement with Programs on a program logic model and with senior management on their information needs

4.0 Findings

4.1 To what extent is the programming addressing a unique and demonstrable need for Atlantic Canadians?

Finding 1: Skills and labour, use of innovative technologies and access to capital are core issues in Atlantic Canada. Needs exist to the same degree or are greater than were reported in the previous evaluation due to increasing demographic pressures.

The results of the evaluation demonstrate a strong, continued need for ACOA's Innovation programming. Innovation is a priority of the federal government and is important for regional economic growth, especially in the context of major demographic and labour market challenges.

Atlantic Canada has been experiencing a well-documented and growing shortage of skills and labour for several decades, which acts as a barrier to the innovation, productivity and growth of SMEs. A review of recent documents and literature reveals that the attraction and retention of talent is expected to continue and be compounded by factors such as out-migration and Atlantic Canada's aging population. A Business Development Bank of Canada (BDC) survey of 1,208 entrepreneurs from across the country showed that hiring employees is more challenging for Atlantic Canadian entrepreneurs than for those in other regions of Canada. In fact, when compared with other regions of the country,

Atlantic Canada had the most (50%) entrepreneurs who reported difficulties hiring in the last 12 months, compared with a national average of 39%. Almost two thirds (62%) expect to face a shortage over the next five years.Footnote (4)

 

The extent of labour shortages in the Atlantic region is surprising because it also has a higher unemployment rate than the Canadian average. Factors contributing to this include the region's older population, lower labour force participation rate and higher share of seasonal workers.Footnote (5) Recent internal analyses suggest that both the labour shortage and the unemployment rate are even more pronounced in rural areas such as Cape Breton (Nova Scotia), northern New Brunswick, and many areas of Newfoundland and Labrador.Footnote (6) The high unemployment rate in some rural areas can be explained by the mismatch between the positions offered and the available labour force (e.g., skills, mobility, etc.).

The need for talent relates to both skilled and unskilled labour, and also includes a gap in business management skills. There is a specific challenge in attracting senior leadership with the skills required to scale up and grow companies beyond the start-up phase. In addition, key informants, the client survey and the literature reviewFootnote (7) underlined that workers with skills in sales (particularly in the technology sector), high-tech/computing, marketing and export are in high demand and difficult to find.

The Atlantic Canadian business landscape is characterized by a need for greater awareness and the use of innovative technologies to improve productivity and growth. There are opportunities to undertake more adaptation and adoption of technology as well as more automationFootnote (8) to address the labour shortage.  Survey results indicate that clients recognize this opportunity: among those clients who indicated that recruiting and retaining employees was a need, 71% indicated their project involved technology acquisition, adaptation or adoption, new equipment or renovations. However, key informants underlined that a risk-averse business culture in Atlantic Canada tends to restrain the use of innovative technologies and challenges the promotion of a growth mindset among entrepreneurs.

Access to funding continues to be a need for Atlantic Canadian firms. Through the ED client survey, 52% (n=493) of ACOA's innovation-related clients indicated that obtaining financing was a key need.  Case studies supported the need for more early-stage financing for start-ups and more angel and venture capital for high-risk ventures in Atlantic Canada. Many key informants underlined that provincial programs in Atlantic Canada do not provide significant direct financial support to SMEs and that this support has declined over the past years.

Some key informants also identified a need for greater access to capital, particularly in the early and growth stages. The perceived gaps were tied to a lack of both private and public-sector investments in Atlantic Canada. A number of key informants suggested that some SMEs might have difficulties accessing large amounts from federal departments such as ACOA and the National Research Council of Canada – Industrial Research Assistance Program (NRC-IRAP) due, at least in part, to a lack of available funding. They noted that access to funding can be particularly challenging for firms with growth trajectories requiring the adoption of very expensive advanced technology or equipment. These challenges speak to the need for the funding continuum available for Atlantic firms to function in a seamless, client-centered way that allows firms access to the financing required to effectively execute their growth plans.

Finding 2: Largely influenced by the program delivery approach, featuring a local presence and relationships with SMEs and partners, the Agency is aware of the varied needs that exist across the region. The Agency has taken some actions to respond to these newer and emerging needs.

The Agency's decentralized delivery model, flexible funding approach, non-financial supports such as pathfinding and convening, along with ongoing and new partnerships and collaborations, all facilitate the Agency's response to changes and opportunities in Atlantic Canada.

ACOA is taking action to address the following items:

  • The skills and labour shortage. Financial support is provided directly to SMEs to permit the hiring and developing of skilled employees. ACOA also contributes to the development of skills and knowledge through its support to non-commercial organizations such as BAIs in Atlantic Canada. Two-thirds (69%) of non-commercial innovation clients indicated that their project increased business skills. Among these, the top four types of skills were marketing, entrepreneurship, business management and sales. One of the key outcomes of the BAIs, as underlined through the case study interviews, is the provision of training, mentorship and a space where start-up entrepreneurs can collaborate and learn from each other. To help address demographic and labour market challenges, ACOA assists employers and communities in attracting and retaining skilled workers to Atlantic Canada. By partnering with the four Atlantic Provinces and Immigration, Refugee and Citizenship Canada (IRCC), the Agency promotes the Atlantic Immigration Pilot (AIP) Program to its clients and refers them to the appropriate partners in the region. Between July 2017 and October 2019, the program granted 5,010 permanent resident admissions to principal applicants and their dependents to settle in the region and fill jobs in a variety of sectors.Footnote (9)

Programming in Action: Atlantic Immigration Pilot

ACOA has assisted Somru Bioscience in P.E.I. since 2013. This biotechnology company develops antibody technology to accelerate the drug development process. The company's vice-president and co-founder immigrated to P.E.I. on a student visa in 2000. The company currently exports to more than 20 countries and is in the midst of a major expansion. Somru is hiring two senior scientists using the AIP to augment local recruiting.

  • The need for greater awareness and the use of innovative technologies. The Agency has increased its focus on advanced manufacturing by providing targeted training to internal employees dealing directly with clients and by funding technology missions for entrepreneurs to learn about new technologies that could lead to increased productivity. A review of recent internal analyses reveals that ACOA's funding for larger-scale automation projects has almost tripled between 2016 and 2019 from $35 million to $95 million. ACOA is also an active participant in the Accelerated Growth Service, an initiative that facilitates a more coordinated approach to supporting established firms with high growth potential. Moreover, the Agency maintains a strong relationship with the NRC-IRAP, an organization dedicated to providing advice, connections and funding to help SMEs drive growth and innovation through applied R&D of technology.
  • The lack of access to capital. ACOA has a flexible approach to providing funding to support specific economic needs across the region; it provided $463 million to innovative Atlantic Canadian companies and to the innovation ecosystem over the period covered by this evaluation. ACOA plays a strong convening and pathfinding role, which helps entrepreneurs access other sources of funding and support.

Finding 3: While ACOA's Innovation programming is unique and complementary to other programming offered in the region, there may be gaps in the funding continuum in Atlantic Canada.

ACOA's programming complements other types of programming and funding available to SMEs and intermediaries for innovation and growth activities. Funding and services for SMEs and other organizations come in a variety of arrangements across the growth spectrum from both government and non-government organizations (Table 7). The main features that set ACOA's programming apart from that of other sources are: mandate to support economic development in Atlantic Canada, focus on SMEs, and flexible financial and non-financial supports.

Table 7: Description of major funding for innovation
Financing Organization Characteristics
ACOA Innovation programming
  • Supports business growth through flexible financial and non-financial contributions for technology development and commercialization
  • Flexibility to support earlier innovation stages (R&D) to help fill funding gaps in the region
  • Repayable and non-repayable
  • Non-dilutive, interest-free
  • Regional presence, localized solutions, active in rural and urban areas
  • Clients include SMEs, BAIs, educational and research institutions, industry associations, other non-profit organizations
National Research Council – Industrial Research Assistance Program (NRC-IRAP)
  • Supports technical innovation through funding contributions, technical and business advisory services and assistance to other government departments Footnote (10)
  • R&D and technology adaptation and/or adoption to the point of pre-commercialization Footnote (11)
  • Non-repayable Footnote (12)
  • Regional presence
  • Co-located with partner organizations including ACOA Footnote (13)
  • Clients include SMEs and not-for-profit organizations that enable services to SMEs
Business Development Bank of Canada (BDC)
  • Federal Crown corporation that supports Canadian entrepreneurs by providing financing (repayable with interest), Footnote (14) advisory services and capital (dilutive) Footnote (15) with a focus on SMEs Footnote (16)
  • Obtain a return on equity equivalent to the government’s cost of funds Footnote (17)
  • Focused on being a complementary lender that fills in the gaps in the lending market Footnote (18)
  • Broader sector coverage including retail, real estate, wholesale
  • Active in rural and urban areas
  • Has initiatives tailored to under-represented groups (e.g., Indigenous, newcomers, women) Footnote (19)
Innovation, Science and Economic Development – Strategic Innovation Fund
  • Financial aid to projects that will improve Canada’s innovation performance while providing economic, innovative and public benefits to Canadians Footnote (20)
  • Supports projects that are requesting at least $10 million Footnote (21)
  • Contributions are generally repayable with some non-repayable options for R&D and innovation ecosystem projects Footnote (22)
Provincial governments
  • Targeted uses, sectors of focus and structuring of financing vary, depending on provincial priorities and availability of funding
  • Arm’s-length organizations, agencies, and Crown corporations (e.g.,  Innovacorp, Opportunities New Brunswick, Innovation PEI, InnovateNL) provide provincial economic development programming
  • Government departments and arm’s-length organizations have experienced changes to type, scope and mandate during the evaluation period Footnote (23)

Finding 4: ACOA's Innovation programming is aligned with current federal and ACOA strategic priorities and contributes to business growth in Atlantic Canada through participation in various initiatives.

ACOA's Innovation programming aligns with the overall federal government objective of helping "Canadian businesses grow, innovate and export so that they can create good quality jobs and wealth for Canadians" as outlined in the mandate letter for the Minister of Innovation, Science and Economic Development (2015). More specific objectives are outlined in several federal priority documents:

  • The mandate letter for the Minister of Innovation, Science and Economic Development (ISED) prioritized the development of an Innovation Agenda that includes expanding support to BAIs, networks and clusters, and working with RDAs to invest in competitive regional advantages.
  • Budget 2016 announced plans to support BAIs, innovation networks, clusters and a tailored approach to coordinating federal services to meet client needs.Footnote (24)
  • Budget 2017 announced the federal Innovation and Skills Plan (2017),Footnote (25) and Budget 2018 announced that it would be delivered by ACOA in the Atlantic region, with support for women entrepreneurs through the Women Entrepreneurship Strategy and support for skills development and economic diversification to help with the transition to a low-carbon economy.Footnote (26)
  • Budget 2018 outlined the new superclusters initiative, including maximizing Canada's ocean economy in Atlantic Canada.Footnote (27) The consolidation of the Accelerated Growth Service and the IRAP Concierge Service was also announced.Footnote (28)

ACOA's Innovation programming aligns with expected results outlined in its DRF under the technology pillar, which aims at seeing businesses investing in the development and commercialization of innovative technologies in Atlantic Canada. A national review of innovation programming announced in Budget 2017Footnote (29) led to revised programming characterized by a stronger focus on innovation, which is a key priority for the federal government and ACOA.

4.2 To what extent is ACOA contributing to business growth in Atlantic Canada?

Finding 5: ACOA's Innovation programming contributes to the growth of new and existing businesses through its supports to the innovation ecosystem, commercialization and productivity improvements.

Overall, this study found that Innovation programming contributes to the achievement of expected outcomes and, ultimately, to the growth of SMEs in Atlantic Canada. ACOA's flexible, non-dilutive funding, delivery model, strategic and collaborative approach, and non-financial supports (e.g., convening and pathfinding) facilitate the achievement of results.

ACOA's funding and other supports contribute to the scale-up and growth of SMEs and a strengthened innovation ecosystem through the commercialization of new or improved technologies, products, processes or services, and productivity improvements. Over the evaluation period, the Agency provided $463 million to commercial and non-commercial clients toward these expected outcomes. For example, internal data shows that the Agency invested $34 million in early-stage SMEs with high potential for growth, universities, and other not-for-profit organizations that provide supports directly to SMEs, such as BAIs, industry associations, and economic development organizations. These projects focused on innovation and commercialization, marketing, scale up, productivity improvements and the development of business skills.Footnote (30)

A strengthened innovation ecosystem:

  • ACOA invested a total of $31 million in BAIs over the period of the evaluation, and has increased its yearly contributions to BAIs and other organizations designed to support innovation of SMEs in the region.
  • Case studies demonstrated the importance of BAIs as spaces for networking and building support capacity.
  • Recent reports from Entrevestor (2017)Footnote (31) and Startup GenomeFootnote (32) underline the importance of ACOA's support to start-ups within an Atlantic Canadian ecosystem increasingly recognized for the availability of early-stage funding. In particular, Atlantic Canada was identified as one of the top 14 ecosystems in the world for "bang for buck" (measured by a ratio of early-stage funding per start-up and average software engineer salary), number one in early-stage funding for activation ecosystem globally (measured by early-stage funding per start-up) and fourth for high-growth activation ecosystem globally (measured by growth in funding, exits and number of start-ups).

Commercialization of new or improved technologies, products, processes or services:

  • ACOA invested $116 million in projects with outcomes focused on commercialization capacity and activities, and $303 million on R&D capacity and activities intended to lead to commercialization.
  • R&D expenditures in Atlantic Canada grew by an annual average of 2.5% from 2005 to 2015, exceeding the national average of 1.6%.Footnote (33)
  • Through the client survey, 75% (n=716) of all ED clients indicated that their project contributed to commercialization.
  • About half (52%, n=476) of commercial, innovation-related clients surveyed indicated that their project improved research capacity (e.g., human resources, facilities, equipment and skills).

Improvements to SME productivity:

  • ACOA strategically invested $22 million in projects aimed at increasing adaptation of technology over the evaluation period.
  • A majority of all clients (69%; n=612) indicated that their project involved technology acquisition, adaptation or adoption.
  • Statistics Canada data show that labour productivity of ACOA-assisted firms experienced a healthy growth compared to non-assisted firms. This includes higher than average annual growth in sales (2.7% versus 0.4%),
  • increased labour productivity (1.7% versus 0.3%) and generation of gross profit at a quicker pace than non-assisted firms.
  • Key informants and published reportsFootnote (34) underline the importance of productivity improvements such as adaptation and adoption of new technologies and automation as part of the solution to the lack of skilled employees.
  • Seventy-one percent (n=346) of innovation-related clients who indicated that recruiting and retaining skilled employees was a need also indicated that their project involved technology acquisition, adaptation or adoption.

Programming in Action: BioNova

ACOA has invested $1.6 million in BioNova, Nova Scotia's life sciences association. Funding assists programs and initiatives to help life sciences member companies grow their business. BioNova also coordinates incoming and outgoing trade activities on behalf of the Atlantic Canada BioIndustries Alliance.

Programming in Action: Commercialization

For over a decade, ACOA has supported Verafin, a Newfoundland and Labrador company that uses artificial intelligence to design and sell innovative financial crime management software. Since its inception in 2003, Verafin has been involved with Memorial University's Genesis Centre and is now considered a leader for other tech start-ups in the community. In 2019, Verafin signed a record-breaking $515 million venture capital deal.

Programming in Action: Advanced Manufacturing

Since 2002, ACOA has provided funding for La Maison Beausoleil in New Brunswick. This company developed a one-of-a-kind automated system that sorts, counts and boxes oysters. This automated production line will enhance productivity and improve product sustainability and food safety.

A complex and changing environment, both internally and externally, affects the achievement of program outcomes. As outlined in section 4.1, the three core needs in the region (e.g., skills and labour, awareness and use of innovative technologies, and access to capital) coupled with the region's risk-averse business culture negatively impact the achievement of results. For example, key informants pointed to the need to support and encourage a growth mindset among business owners, due to a reluctance to take on riskier projects that may contribute to growth. The Agency's use of provisionally repayable contributions aid clients in undertaking these riskier projects, as businesses need access to more risk capital that is not available through traditional lenders.

Other factors, more closely within the control of ACOA, were also raised through key informant interviews and published reports as challenging the achievement of outcomes.  

Finding 6: The absence of programming would have had a major negative impact on the achievement of outcomes.

ACOA's financial investments directly lead to the achievement of project results and allow projects to proceed as planned, including through the leveraging of other funding. Case studies, key informant interviews and client survey results reflect that ACOA supports have a major impact on the Atlantic Canadian innovation ecosystem.

ACOA's support facilitates leveraging of other sources of funding. Case studies indicate that ACOA's involvement and investments have a major influence on leveraging from other sources, as the Agency is perceived as having solid project risk assessment, monitoring and reporting processes.

The need for ACOA supports exists to at least the same extent as it did in the previous evaluation of the programming, completed in 2015. The importance of ACOA's contributions is supported by the results of the client survey with 91% (n=496) of innovation-related clients indicating that their project would have experienced a major negative impact without ACOA's support. As detailed in Figure 1, if the projects had still gone ahead, most innovation-related clients would have experienced delays or projects with a smaller scope. Case study results indicate that a majority of BAIs in the region, important actors in the innovation ecosystem, would not exist without ACOA support.

The need for ACOA supports exists to at least the same extent as it did in the previous evaluation of the programming, completed in 2015. The importance of ACOA's contributions is supported by the results of the client survey with 91% (n=496) of innovation-related clients indicating that their project would have experienced a major negative impact without ACOA's support. As detailed in Figure 1, if the projects had still gone ahead, most innovation-related clients would have experienced delays or projects with a smaller scope. Case study results indicate that a majority of BAIs in the region, important actors in the innovation ecosystem, would not exist without ACOA support.

Client survey – Impacts of no ACOA support : long description

Figure 1 details, in a bar chart, clients’ perception of what would have happened to their project(s) without ACOA’s support, including that their project(s) would have been delayed or the completion time extended (68%), their project(s) would have had a smaller scope (56%), their project(s) would have had a higher level of risk (43%), they would have assumed more debt (39%), they would have looked for private capital/investment (25%), they would have lost partners (13%), they would have experienced other impacts (4%), and/or they did not know (3%).

4.3 To what extent is ACOA taking a client-centric approach to program design and implementation?

Finding 7:  The design and delivery of Innovation programming supports a client-centric approach to some extent. Opportunities exist to take a more strategic approach to program delivery and portfolio risk.

Recent changes to program funding mechanisms are perceived as positive for the ecosystem. According to key informants, the implementation of the REGI program in 2018-2019 was relatively seamless and did not create issues for clients or for ACOA staff. Furthermore, key informants reported improved clarity around eligibility of projects without affecting the ability to fund projects flexibly. However, program changes over the evaluation period raised questions about the role of the AIF. Given new programming and other factors, some key informants identified the need to rethink the AIF's funding objectives and mechanisms to address possible overlap with REGI and gaps in client needs.

While ACOA's supports for the innovation ecosystem correspond to client needs, some gaps continue to exist in coordination and communication. Many key informants noted the important work that ACOA has done to improve BAI coordination in recent years. They also stated that ACOA could stand a more coordinated and consistent approach to funding BAIs to improve their ability to pivot strategically, collaborate and avoid duplication of efforts. They also stated that better coordination, collaboration and knowledge sharing between regional offices is needed while balancing the need for flexibility to meet different needs across the region.

Some ACOA key informants noted that recent internal coordination mechanisms (e.g., champion files) facilitate pan-Atlantic collaboration between ACOA's offices and improve understanding of client needs. Some suggested that ACOA's existing governance structures do not always promote the more strategic, matrix approaches. Internal key informants also report that there is a need to clarify language and definitions of certain key themes related to newer priorities to promote better cohesion and understanding among ACOA staff (e.g., innovation, start-up, scale up).

ACOA provides funding to a range of non-commercial organizations that are actors in the innovation ecosystem and that help to address SMEs' needs for advice, access to funding and other supports. For example, key informants indicated that support offered by ACOA to most Atlantic Canadian BAIs provides start-ups with access to tailored programming based on industry needs in communities across the region.

ACOA's ability to meet client needs may be impacted by its risk tolerance level. There is a perception that ACOA's culture remains risk averse. Key informants report that the Agency's approach to risk is complex and can affect the types of projects developed and approved as well as levels of funding. Key informants indicated a need for the Agency to increase the risk level of its portfolio to further support the growth of SMEs and encourage more risk within the ecosystem. A faster, more agile and certain Atlantic Canada states that innovation requires risk tolerance and that governments should tolerate a higher acceptable level of risk in order to increase overall return on investments.Footnote (36)

Examples of actors in the Atlantic Canada innovation ecosystem:

  • Commercial businesses
  • Business Accelerators and Incubators (BAIs) – support for start-ups and scale-ups
  • Universities and research institutions – R&D
  • Other non-commercial organizations – e.g., Springboard, BioNova, industry associations
  • Government departments and agencies

ACOA's approach to risk will be examined more closely in the second report resulting from this evaluation – the Evaluation of ACOA's Business Growth and Trade Programming – expected to be finalized in May 2020.

Finding 8: ACOA contributes to increased collaboration among stakeholders and clients. However, opportunities exist to improve coordination and collaboration within the innovation ecosystem.

ACOA's non-financial supports contribute to meeting client needs. Most surveyed innovation-related clients (89%; n=470) indicated satisfaction with the business knowledge and advice offered by ACOA. Key informants stated that the Agency has worked to increase the capacity of its staff in areas that are critical for the region, such as advanced manufacturing, automation and artificial intelligence. They also underlined that ACOA's convening role, including through traditional project supports and through the Accelerated Growth Service, was key in supporting innovation and growth.

ACOA is playing a more deliberate convening and pathfinding role since the previous evaluation of the programming. In addition to important funding, the Agency provides non-financial support such as advice, guidance, expertise or assistance to clients and communities. Innovation-related client survey results revealed that this type of support is considered important by 83% (n=495) of respondents. Some clients explained that ACOA helps them connect with other players and provides visibility to their project, while others said that the Agency's personnel made their company aware of funding opportunities, which led to obtaining additional capital. Key informants and case studies indicated that a more formalized approach to convening and pathfinding roles is also taking place through the Accelerated Growth Service and coordination of BAIs. As detailed in Figure 2, a variety of partners were involved (either through provision of financial or in-kind support) in ACOA-assisted projects (n=291)

Client survey – Organizations involved in projects beyond ACOA : long description

Figure 2 details, in a bar chart, clients’ perception of the other organizations involved in their project(s) beyond ACOA, including provinces (65%), SMEs (48%), corporations (43%), academics (35%), financial institutions (34%), research institutions (34%), industry associations (34%), municipalities (21%), other NGOs (21%), BAIs (20%), Indigenous communities (12%), other organizations (6%), and/or other federal departments (4%).

Key informants, case studies and published reports (e.g., BAI Performance Measurement Framework by the Deep Centre)Footnote (37) point to the growing engagement of ACOA with other federal partners such as Employment and Social Development Canada, the Department of Fisheries and Oceans Canada, Global Affairs Canada and IRCC (through the Atlantic Immigration Pilot). ACOA's support to the innovation ecosystem through BAIs also promotes collaboration within the innovation ecosystem itself. BAIs play a key role in providing space for collaboration and networking among entrepreneurs, as revealed through the case studies. In some regions, co-location between ACOA and the NRC-IRAP was also underlined as a best practice, improving collaboration and complementarity between the two organizations.

In light of the growth and evolving nature of the ecosystem, there are opportunities to improve coordination of partners and actors. For example, continued and enhanced coordination between BAIs and ACOA's regional offices could be beneficial to avoid risk of duplication within the same region and to promote sharing of best practices among Atlantic Canadian BAIs.

SpringboardFootnote (38), an organization funded by ACOA to work with researchers and industry to develop relationships that will lead to commercialization, is an example of a useful mechanism for collaboration in Atlantic Canada. An ACOA internal review of Springboard conducted in 2018 found that while progress has been made on developing better connections between researchers and industry to support innovation

and commercialization, there are opportunities to improve coordination and communication by tailoring research strategies to needs of business, improving engagement and strengthening relationships with industry. As a result of the review, ACOA's new funding agreement with Springboard (2020-2023) will see the organization build upon its established network to increasingly focus on projects and plans that support regional strategic objectives. While Springboard will continue the momentum that has been generated throughout previous funding agreements relative to collaborative industry research engagements and commercialization support, an increased emphasis will be directed toward supporting a robust industry engagement and technology commercialization network; advancing innovation in strategic industries and sectors and facilitating the growth of start-ups and high-potential firms.

Programming in Action: Innovation Ecosystem

ACOA has supported Volta Labs Inc. in Nova Scotia since 2013. This not-for-profit entity has strengthened the technology ecosystem in Atlantic Canada through mentorship and community-driven aid. Volta Labs' programs include Volta Cohort, a collaboration with ACOA, the Business Development Bank of Canada, and Innovacorp, to help early-stage, high-calibre companies from across Atlantic Canada attract and secure investment.

Finding 9: There is a perception that programs are open to all equally, but there is growing recognition that there are opportunities to better support specific groups.

While ACOA has long delivered programming for particular population groups, it has paid increasing attention to certain vulnerable populations over the evaluation period. ACOA is particularly strong in delivering programming to meet the needs of rural populations and has made strides to increase efforts for new immigrants, Indigenous people and women. No specific focus on youth was noted through this evaluation.

On July 4, 2016, the Government of Canada and the four Atlantic Provinces launched the Atlantic Growth Strategy, which sought to drive long-term economic growth in the Atlantic region by implementing targeted, evidence-based actions under five priority areas.Footnote (39) The Agency champions immigration mostly through activities in partnership with IRCC. ACOA's involvement in the Atlantic Immigration Pilot (2017-2020), a key initiative under the Atlantic Growth Strategy, helps address labour and skills needs and ensures long-term economic growth in the region by attracting and retaining skilled immigrants.Footnote (40) The case studies also underlined that BAIs supported by the Agency provide supports to immigrant entrepreneurs, although retention remains a challenge. Based on Statistics Canada data, there is room for improvement, as immigrants owned a low share (5.2%) of ACOA client firms and comparable firms (5.1%) in 2015.Footnote (41)

ACOA has recently prioritized Indigenous economic development through the creation of a pan-Atlantic priority file, but there remains an opportunity to provide more support for indigenous businesses. In Atlantic Canada in 2017, 1.1% of all businesses had indigenous ownership and that percentage has decreased compared to 2014. The Atlantic Province Economic Council Indigenous report underlines specific training needs related to networking, building and maintaining relationships with financial institutions, advice on business growth, access to funding, entering new markets, developing a business strategy, and finding and retaining skilled employees. Key informant interviews indicated that more could be done in terms of adapting ACOA's approach to Indigenous development, but acknowledged that efforts are being made through the Agency's Indigenous file. Some promising practices are emerging, however, such as recent efforts in Cape Breton to develop strong partnerships and relationships with Indigenous communities.

The programming also includes some initiatives for women in business. Key informants and case studies indicated that a number of initiatives are in place to increase women's presence in the business and innovation space. ISED's Women Entrepreneurship Strategy, a national $2 billion investment put in place through Budget 2018,Footnote (42) aims to help women start or grow a business. Other initiatives are undertaken through the BAIs in Atlantic Canada, such as Genesis' Women in Tech Peer Group taking place in Newfoundland and Labrador.

Statistics Canada data and key informants suggested that further efforts would be merited. Women owned only 11.9% of ACOA-assisted firms in 2015 compared to 10.8% in 2010.Footnote (43) In the innovation ecosystem, women-led start-ups receive a lower share of venture capital in Atlantic Canada compared with the rest of Canada.Footnote (44) Despite this, between 2010 and 2015, female ACOA clients not only registered higher average growth rates than male ACOA clients in a number of employment and financial indicators but also outperformed female-owned businesses from the comparable group in most areas. Case studies indicated that there has been increased focus on supports to women in technology, computer science and entrepreneurship.

Atlantic Canada is highly rural and unique challenges exist for businesses outside urban centres. Key informants and key documents indicated that rural areas experience challenges, in particular related to access to risk capital for start-ups.Footnote (45) However, key informants indicated that ACOA's 28 points of service, including many in rural communities, facilitate meeting specific client needs. The Agency's flexible approach also allows for support tailored to rural-based companies.

Finding 10: There is evidence to suggest that recent internal operational changes are supporting more efficient program delivery.

Delivery costs for ACOA's Innovation programming are comparable to amounts reported in previous evaluations. The total expenditures for Innovation programming included in this evaluation were $507.6 million. An analysis of internal financial data indicates that ACOA's cost to deliver Innovation programming has remained relatively unchanged over time, with a cost to deliver $1.00 of G&C funding of $0.097 over the period of this evaluation (2013-2014 to 2018-2019), compared with $0.092 as reported in the last evaluation (2007-2008 to 2012-2013).

There is a high degree of satisfaction with ACOA's service features. Key informants and case studies indicated that ACOA staff are proactive, efficient and generally helpful at all stages of the project life cycle, including completing forms and navigating the application and claims processes. They described ACOA as the cornerstone of many projects and stressed the importance of non-financial supports provided by ACOA staff. As detailed in Figure 3, the overwhelming majority of innovation-related client survey respondents indicated strong satisfaction with a range of ACOA's service features. They are very satisfied with the courteousness and professionalism of ACOA personnel (97%), the ongoing business relationship with ACOA personnel (94%), communication from ACOA personnel on the status of their request (93%), and the availability of ACOA personnel (93%).

Client survey – Satisfaction with ACOA service features : long description

Figure 3 details, in a bar chart, the degree of client satisfaction with ACOA client service features, ranging from very satisfied to very dissatisfied.

Project approval times have improved, though some clients still see them as a challenge. When compared with other ACOA processes and service features, innovation-related client survey results revealed a slightly lower satisfaction rating for ease of the application process and paperwork (84%) and speed of turnaround time of the application process (82%). Internal key informants noted that recent operational changes focused on streamlining processes, including changing delegation levels, have improved approval times. The transition to the redesigned PAF also facilitated programming efficiency. In 2017, the Agency undertook a pilot project to examine potential areas for improving and streamlining project evaluation and approval processes, with a specific focus on commercial applications. The goals of this exercise were to minimize the burden on clients by expediting processing of applications and reducing wait times, and establishing service standards for specific steps in the approval process.

5.0 Conclusions

Overall, the evaluation found that ACOA's Innovation programming remains relevant and achieves expected outcomes related to economic development in Atlantic Canada. The delivery model is grounded in the Agency's deep knowledge of businesses and expertise in convening and pathfinding, which facilitates the achievement of results. There are some areas for attention to ensure ongoing and strengthened strategic investments through the programming. The evaluation concludes that:

  1. Programming is relevant to the existing needs of SMEs, aligned with ACOA's mandate and GC priorities, and complements other supports in the region.
  2. ACOA is aware of and has addressed changing needs to a certain extent. There are risks to its continued ability to address the needs of SMEs due to potential funding gaps, access to labour and skills, and uncertainties about AIF funding.
  3. ACOA's Innovation programming contributes to the growth of new and existing businesses through its supports to the innovation ecosystem, commercialization and productivity improvements. ACOA plays an important role in developing and maintaining collaborations, pathfinding and convening. While there are opportunities to improve the ability to capture and report on longer-term impacts of the programming, the current implementation of a new project management system and other initiatives under way hold promise in improving the quality and availability of performance measurement data, including that related to BAIs.
  4. The design and delivery of programming supports awareness of changing needs, a client-centric approach and the achievement of results. A strong regional presence allows for knowledge of needs, capacities and partners as well as the development of relationships across the ecosystem. The recent implementation of REGI appears to be positive. Opportunities exist to further improve internal coordination and to better address client needs, including those of under-represented groups.
  5. ACOA has mechanisms in place to support the efficient delivery of programming to facilitate the achievement of outcomes. Opportunities exist to improve the direction on appropriate risk tolerance levels, to clarify the direction for the AIF, and to build upon existing internal and external coordination and collaboration efforts.Footnote (46)

6.0 Recommendations

Within the context of the recent ACOA funding program changes, and the complexity that exists in the innovation ecosystem in Atlantic Canada, ACOA should:

Recommendation 1: Build upon current Agency efforts to address any gaps in funding that may impact the scale up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners.

Recommendation 2: Examine AIF funding in relation to the current needs of SMEs and priorities of the Government of Canada to confirm the strategic direction of the program and any changes to governance and funding processes.

Recommendation 3: Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes and better coordination among stakeholders while recognizing unique regional and SME needs.

Recommendation 4: Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration.

Appendix B presents a one-page summary of key findings, conclusions and recommendations.

ACOA senior Programs management has agreed with the evaluation's recommendations. It has developed a MAP that details the actions the Agency will take to address each of the recommendations. Appendix C presents the MAP.

Appendix A: Innovation programming logic model

Appendix A : long description

Appendix A details the Innovation logic model, which provides an overview of the program’s principle inputs, activities and outputs as well as expected immediate, intermediate and ultimate outcomes.

Appendix B: Summary of findings, conclusions and recommendations

FINDINGS CONCLUSIONS RECOMMENDATIONS
Relevance
  1. Programming is relevant to the existing needs of SMEs, aligned with ACOA’s mandate and GC priorities and complements other supports in the region.
  2. ACOA is aware of and has addressed changing needs to a certain extent. There are risks to its continued ability to address the needs of SMEs due to potential gaps in the funding continuum, SMEs’ access to labour and skills, and uncertainties about AIF funding.
  3. ACOA’s Innovation programming contributes to the growth of new and existing businesses through its supports to the innovation ecosystem, commercialization and productivity improvements. ACOA plays an important role in developing and maintaining collaborations, pathfinding and convening. While there are opportunities to improve the ability to capture and report on longer-term impacts of programming, the current implementation of a new project management system and other initiatives under way hold promise in improving the quality and availability of performance measurement data, including that related to BAIs.
  4. The design and delivery of programming supports awareness of changing needs, a client-centric approach and the achievement of results. A strong regional presence allows for knowledge of needs, capacities and partners as well as the development of relationships across the ecosystem. The recent implementation of REGI appears to be positive. Opportunities exist to further improve internal coordination and to better address client needs, including those of under-represented groups. Footnote (47)
  5. ACOA has mechanisms in place to support the efficient delivery of programming to facilitate the achievement of outcomes. Opportunities exist to improve the direction on appropriate risk tolerance levels, to clarify the direction for the AIF, and to build upon existing internal and external coordination and collaboration efforts.
Within the context of recent ACOA funding program changes, and the complexity that exists in the innovation ecosystem in Atlantic Canada, ACOA should:
  1. Build upon current Agency efforts to address any gaps in funding that may impact the scale-up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners. (B, D).
  2. Examine AIF funding in relation to the current needs of SMEs and priorities of the GC to confirm the strategic direction of the program and any changes to governance and funding processes (B, E).
  3. Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes, and better coordination among stakeholders while recognizing unique regional and SME needs (C, D, E).
  4. Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration (B, D).
Needs related to innovation in Atlantic Canada exist to the same degree or are greater than reported in the previous evaluation. Shortage of skills and labour is increasingly a challenge influenced by demographic shifts; awareness and use of innovative technologies and access to specific types of capital (early stage, growth stage, angel and venture capital) are also important.
The program delivery model with local presence and convening and pathfinding roles allows the Agency to be responsive to diverse and changing economic needs across the region. A national review of innovation programming led to the implementation of new programming (e.g., the REGI program). ACOA developed or strengthened collaborations to better address immigration, skills and labour, and funding.
Programming offers unique features and is complementary: flexible, non-dilutive funding options and non-financial supports focused on promoting growth of SMEs in Atlantic Canada. Key informants suggest gaps in the spectrum of federal funding exist and are influenced by unique needs in Atlantic Canada.
ACOA’s Innovation programming is aligned with federal government priorities on innovation and business growth. It is clearly aligned with ACOA’s mandate to lead economic development in the region through supports to SMEs.
Contribution to Business Growth
ACOA investments support innovation in Atlantic Canada ($463 million to 850 projects over six years). ACOA clients report higher R&D and productivity compared to non-supported firms (Statistics Canada). In a survey of ACOA clients, 77% reported that projects contributed to commercialization; 75% indicated that their projects improved the efficiency or capacity of operations; 69% said that it involved technology acquisition, adaption or adoption, new equipment or renovations. Key informants, case studies and a document review indicated that ACOA contributes to a strengthened ecosystem.
ii
The absence of programming would have had a major negative impact on the achievement of results. Ninety-one percent of clients said the absence of ACOA funding would have had a major negative impact on their project; 48% of commercial clients and 80% of non-commercial projects would not have proceeded without ACOA support. ACOA’s support facilitates the leveraging of other sources of funding: ACOA leverages $1.22 from partners per $1 of funding. Key informants and a literature review revealed that the start-up ecosystem (e.g., BAIs) would be weaker without ACOA supports.
Client-centric Approach
ACOA employs mechanisms to focus on client needs: provision of funding and non-financial supports; program flexibility and delivery model; internal coordination on priorities; partnerships and collaborations that leverage funding. Early implementation of REGI has been positive, with no major impacts on the ability to fund projects and improved internal clarity on eligibility. Factors that impede client focus to some degree: program uncertainty during the period (e.g., transition to REGI, AIF); ACOA’s portfolio risk tolerance; aspects of internal and external coordination; broad economic challenges (e.g., demographics, capital, business culture). Key informants suggest that strengthened coordination with key partners may decrease the risks of funding gaps.
ACOA contributes to increased collaboration among stakeholders and clients through both funding for projects and other activities. Eighty-three percent of clients reported that ACOA’s non-financial supports are important. ACOA funds certain non-commercial organizations that support collaboration (e.g., Springboard, BioNova, BAIs). A document review and key informants indicated that recent efforts to improve coordination among BAIs are positive and must continue. ACOA is playing an expanded and more deliberate role in convening and pathfinding and has developed new partnerships to address changing priorities.
The Agency is making some efforts towards inclusive growth (e.g., immigration, Indigenous, women). However, there is growing internal recognition of the need to better support under-represented groups to facilitate economic growth, address skills and labour gaps, and better align with government priorities.
The cost to deliver $1 of G&C funding was $0.097 and aligned with costs in other program areas. Recent internal operational changes are supporting more efficient program delivery (e.g., PAF, streamlined approval process, and planned client information management system). A client survey revealed a high degree of satisfaction (89%) with ACOA service features. Key informants and a review of data identified questions related to efficiency: project approval times, performance data, ACOA’s risk tolerance levels.

Appendix C: Management action plan

RECOMMENDATION PLANNED ACTIONS RESPONSIBILITY TARGET DATE
Recommendation 1:
Build upon current Agency efforts to address any gaps in funding that may impact the scale-up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners.
The Agency will coordinate with other federal innovation partners in Atlantic Canada to explore issues impacting SME financing and to ensure Atlantic Canadian firms have access to funding to support their growth. DG Programs (HO), supported by DG OPS July 31, 2020
Recommendation 2:
Examine AIF funding in relation to the current needs of SMEs and priorities of the GC to confirm the strategic direction of the program and any changes to governance and funding processes.
The Agency will undertake an examination of AIF funding in relation to the current needs of SMEs, other innovation programming, and the priorities of the GC. DG Programs (HO), supported by DG OPS July 31, 2020
Recommendation 3:
Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes, and better coordination among stakeholders while recognizing unique regional and SME needs.
Support the National Performance Measurement Framework for BAIs to ensure that organizations receiving ACOA funding are assessed according to standard criteria, consistent with national expectations, thus enabling the Agency to better link funding decisions to expected outcomes.
Facilitate opportunities that continue to foster strengthened collaborations among regional stakeholders.
DG Operations NS on behalf of the ACOA Start-up Champion, supported by DG Programs (HO) March 31, 2021
Recommendation 4:
Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration
Continue ongoing efforts to engage federal, provincial, institutional and industry collaborators in pursuing strategies that improve the efficacy of the labour and skills ecosystem, which includes funding pilot projects, research into evolving trends and previous Agency expenditures, convening relevant stakeholders, and advocating the specific interests of Atlantic Canada to increase the quantity and quality of the labour force to meet current and future industry demands. VP NB, supported by DG Programs (HO) March 31, 2021

Appendix D: Innovation evaluation framework

Note: A bolded "L" refers to the leading line(s) of evidence. A small "s" refers to supporting line(s) of evidence.

Evaluation questions   Judgment criteria / Indicators Existing information New information
Literature Review Document Review Performance Data (Stats Can) Performance Data (Internal) Financial data (GX) Interviews Survey Case Studies
1. To what extent is the programming addressing a unique and demonstrable need for Atlantic Canadians?
  • What are the economic development needs in Atlantic Canada? To what extent does ED programming respond to these economic development needs?
  • Nature and evolution of needs
  • Evidence the program is aligned with and responds to the varied needs that exist across regions
  • Stakeholder perceptions that programming responds to needs
L L       s s s
  • To what extent is the programming unique and complementary to other programming offered in the region?
  • ACOA’s ED programming complements rather than duplicates or overlaps other programming, including that of other federal and provincial governments
  • Stakeholder perceptions on complementarity and shifting roles in ecosystem
L s       s s L
2. To what extent is ACOA contributing to business growth in Atlantic Canada?
  • How has ACOA contributed to key expected longer-term outcomes: strengthened innovation ecosystem, increased commercialization, increased SME productivity, and increased exports and investment?
  • What factors facilitate or impede the achievement of outcomes?
  • Nature and impact of innovation investments on the achievement of results through evidence on investments in:
    • BAIs, start-ups, SMEs for innovation activities
    • Applied research
    • Development of new or improved technologies, products, processes or services
    • Commercialization
    • Acquisition, adaptation and adoption of technology
    • New equipment or renovations
  • Evidence that outcomes are being achieved to a level at or above established targets
  • Evidence that outcomes are being achieved to a level similar to or greater than that observed in the previous evaluation (taking into consideration changes in context, processes and procedures)
  • Evidence supporting program areas’ contribution to the achievement of program outcomes and current priorities
  • Evidence that the outcomes of ACOA-supported firms are equal to or greater than those of comparable firms
  • Stakeholder perceptions on program impacts
  • Evidence of increased SME growth (revenues, sales, labour, profit, productivity)
  L L L s s L s
  • How does the program align with current federal government and ACOA strategic priorities? To what extent do investments in newer priorities contribute to business growth in Atlantic Canada?
  • Extent of alignment between current federal priorities and Innovation programming
  • Extent of alignment between current ACOA priorities and Innovation programming
  • Evidence that ACOA’s investments in newer priorities contribute to expected program outcomes
  • Stakeholder perceptions on early impacts of investments in newer priorities
  L   L s s L s
  • What impact would the absence of program funding have on ED in Atlantic Canada?
  • Evidence about extent to which the impact of the absence of the programming is similar to or greater than that observed in previous evaluations
  • Stakeholder perceptions on program impacts, that is, clients attribute, at least to a moderate extent, the achievement of outcomes to ACOA’s support
  L s s s s L L
3. To what extent is ACOA taking a client-centric approach to program design and implementation?
  • To what extent have the design and delivery of the program addressed the business growth needs of clients in the ecosystem?
  • What have been the nature and early impacts of program funding changes?
  • To what extent has the approach to risk changed over time?
  • To what extent has ACOA employed a corporate strategic approach to economic development?
  • What factors facilitate or impede efficient program delivery?
  • Stakeholder perceptions of how well the program meets clients’ business growth needs
  • Client satisfaction with business knowledge and advice offered by ACOA staff
Program funding changes, evidence of:
  • Coordination among new and ongoing funding programs
  • Any impact on results and efficiency from the move away from early R&D funding
Strategic approach, evidence of:
  • Programming oversight includes: documented governance, priority setting, planning and decision making, as well as use of performance measurement and risk management
  • ACOA’s approach to program delivery is client-centric and enables SMEs to access a range of services across ED programming as required
  s   s s L L s
  • To what extent has ACOA contributed to increased collaboration (convening, pathfinding, leveraging) among stakeholders and clients? Best practices? Lessons learned?
  • Amount leveraged per dollar invested by ACOA
  • Number of projects with partners and the nature of these collaborations
  • Evidence of changes in ACOA’s role and activities related to collaboration (convening, pathfinding, leveraging) over the last five years
  • Stakeholder perceptions of how ACOA’s investments influence the involvement of other partners
  s   L s L s L
  • To what extent and how does the program design and implementation contribute to inclusive growth (language minorities, youth/seniors, Indigenous peoples, new immigrants, gender, rural/urban, etc.)?
  • Evidence that population differences are reflected in program design, including program theory, objectives and outcomes
  • Evidence that the program delivery includes collaboration and engagement with diverse groups
  • Evidence of program impacts on under-represented/marginalized groups – women, francophones, Indigenous, youth, and new immigrants
  • Stakeholder perceptions of contribution to inclusive growth
  s s s s L s L
  • To what extent have recent internal operational changes resulted in more efficient program delivery (program simplification, Agency-wide teams for newer priorities, capacity of account managers, integration of GBA+)?
  • Delivery costs have remained stable or decreased since last evaluation
  • Stakeholder perceptions of how program delivery has changed over time, current level of efficiency and satisfaction
  s   s s L s L

Appendix E: Enterprise Development evaluation terms of reference overview

Appendix E : long description

Appendix E details the terms of reference for the ED evaluation including an overview of the evaluation scope, approach, methods, timeline and governance. 

Appendix F: Evaluation advisory committee

Appendix F : long description 

Appendix F details the evaluation advisory committee including an overview of the evaluation’s governance, roles and responsibilities, engagement and timelines.

Page details

Date modified: