Evaluation of ACOA’s innovation and commercialization sub-program - final report
Evaluation Unit, Evaluation and Risk Directorate
Atlantic Canada Opportunities Agency
January 26, 2015
Glossary of acronyms
ACOA Atlantic Canada Opportunities Agency
AIF Atlantic Innovation Fund
BDC Business Development Bank of Canada
BDP Business Development Program
BERD business enterprise research and development
CBoC Conference Board of Canada
CIHR Canadian Institutes of Health Research
DG director general
ED Enterprise Development
FTE full-time equivalents
G&C grants and contributions
GDP gross domestic product
IBD International Business Development
IC Innovation and Commercialization
IRAP Industrial Research Assistance Program
NRC National Research Council of Canada
NSERC Natural Sciences and Engineering Research Council
O&M operations and maintenance
OECD Organization for Economic Co-operation and Development
PAA program alignment architecture
PG Productivity and Growth
R&D research and development
RDA Regional Development Agency
SME small and medium-sized enterprise
SSHRC Social Sciences and Humanities Research Council of Canada
Table of contents
1.2 Evaluation Approach and Methodology
1.3 Evaluation Challenges and Mitigation Measures
2.0 Profile of the Innovation and Commercialization Sub- program
3.1 Continued Need for the Programming
3.2 Relationship to Other Programming
3.3 Alignment with Federal Priorities, Roles and Responsibilities
4.0 Findings: Performance – Effectiveness
4.1 Investment in Innovation and Commercialization Projects
4.2 Achievement of Expected Outcomes
4.3 Contributing and Constraining Factors and Lessons Learned
4.4 Adequacy of Performance Measurement
5.0 Findings: Performance – Efficiency and Economy
5.1 Efficient Utilization of Resources
5.2 Structures and Mechanisms that Support Efficiency and Economy
5.4 Alternative Modes of Delivery
6.0 Conclusions and Recommendations
Appendix A: Management Action Plan
Appendix C: Evaluation Issues and Questions
Appendix D: Innovation and Commercialization Sub-Program Logic Model
Appendix E: Other Programs Supporting Innovation and Commercialization in Atlantic Canada
Appendix F: Client Survey Ratings Tables
Appendix G: Profile of IC Projects, by Program and Type of Client, 2005-2006 to 2012- 2013
List of figures
Figure 1: Overview of Leading Programs in Atlantic Canada
Figure 2: Primary Focus of the Projects Surveyed, by Program Area
Figure 3: Partnerships Between Institutions and Private
List of tables
Table 1: Response Rate, by Program and Type of Client
Table 2: Profile of ACOA Assistance, by Program and Type of Client, 2005-2006 to 2012- 2013
Table 3: Operations and Maintenance Expenditures, 2005-2006 to 2012-2013
Table 4: FTEs Dedicated to Innovation and Commercialization, 2007-2008 to 2012-2013
Table 5: Immediate Impacts of IC Sub-program Projects
Table 6: Intermediate Outcomes of IC Sub-program Projects
Table 7: Long-Term Outcomes of IC Sub-program Projects
Table 8: Program Delivery Costs by Fiscal Year
Table 9: Selected Results from the 2010 and 2014 Evaluations
Table 10: Comparison of Delivery Costs, ACOA’s IC Program and NRC-IRAP Program
Table 11: Importance of Factors When Considering Applying to ACOA versus Another Source
Table 12: Ratings of Immediate Impacts Generated by Undertaking the Project
Table 13: Ratings of Intermediate Impacts Generated by Undertaking the Project
Table 14: Ratings of Long-term Impacts Generated by Undertaking the Project
Table 15: Use and Ratings of Other Resources Supported by ACOA
Acknowledgements
This evaluation provides the Atlantic Canada Opportunities Agency (ACOA) management with systematic, neutral evidence on the relevance and performance of its Innovation and Commercialization sub-program. It builds on a recent internal review of the programming and supports ongoing decision making and monitoring of this critical element of ACOA’s portfolio.
The evaluation was designed and led by ACOA’s Evaluation Unit, and the study used a hybrid team approach involving ACOA’s internal evaluators and external consultants from Ference Weicker & Company. We thank the consulting team for their close collaboration, completion of primary research, synthesis, integration and reporting. We also thank the members of the evaluation advisory committee for their advice and support throughout the evaluation process. Their involvement and assistance helped ensure that the study and its recommendations are both relevant and useful. Of special note are the gracious contributions of three external advisory committee members and one external advisor: Rebecca Hart, Evaluation Officer, National Research Council of Canada; Dr. Jacquelyn Thayer Scott, Professor of Organizational Management and Public Administration, Cape Breton University and Chair, Board of Directors, Innovacorp; Peter Crooks, Executive Director, Canada’s Smartest Kitchen, Holland College; and Dr. Harald Bathelt, Canada Research Chair in Innovation and Governance, University of Toronto (external advisor).We are also grateful to the many ACOA staff members, including members of the Evaluation Unit, ACOA clients and external key informants who provided their time and essential knowledge in support of this study.
Overall, these contributions were instrumental in building our understanding and identifying best practices and opportunities for continuous improvement for ACOA’s Innovation and Commercialization programming.
Courtney Amo
Director, Evaluation and Risk Directorate (Head of Evaluation)
Atlantic Canada Opportunities Agency
Executive summary
The evaluation focuses on Innovation and Commercialization (IC) sub-program activities approved between 2005-2006 and 2012-2013. During this period, $652.5 million in ACOA assistance was approved for 1,131 IC projects. Operations and maintenance expenditures (O&M) totalled $56.2 million.
The IC sub-program is part of the Atlantic Canada Opportunities Agency’s (ACOA) Enterprise Development (ED) Program and is delivered through two grant and contribution (G&C) programs: the Atlantic Innovation Fund (AIF) and the Business Development Program (BDP). The AIF and BDP target both commercial and non-commercial entities (including research institutions, universities and other not-for-profit organizations). The AIF focuses on increasing research, development and innovation in Atlantic Canada and, for the period under review, was delivered through an Agency-wide competitive process. The BDP is delivered regionally and has a broader scope, encompassing a variety of eligible activities that support the growth and competitiveness of small and medium-sized enterprises (SMEs). ACOA recently revised its IC programming by launching a new BDP commercialization funding element, streamlining the intake of applications, changing the AIF process to a continuous intake system, and lowering the minimum funding threshold to better enable the AIF to meet business needs.
The evaluation of the IC sub-program is expected to contribute to learning, future decision making, and accountability (meeting the Treasury Board Policy on Evaluation requirement that all direct program spending be evaluated every five years). The evaluation employed multiple lines of evidence, including surveys of 277 IC clients, 16 case studies involving 52 projects, interviews with 60 key informants, a document and literature review, a review of project and operational data, the development of regional profiles and an economic impact analysis.
Conclusions
Relevance
- There is a strong, continued need for the IC sub-program given the importance of innovation and commercialization to regional competitiveness, the need to build on the progress achieved in recent years, and the fit between the IC sub-program and the needs of clients. Innovation and commercialization are major drivers of economic competiveness. Atlantic Canada, and more specifically the IC sub-program, has made progress in strengthening commercialization capacity and increasing investment in research and development (R&D) over the past decade, yet the region continues to lag behind other regions of Canada on key innovation and commercialization indicators (e.g. level of investment in R&D, level of R&D performed by the private sector, access to venture capital and highly qualified personnel, proportion of federal research grant funding awarded, etc.). The IC sub-program responds to this need by providing support for innovation and commercialization projects that meet the needs of clients and would otherwise not proceed as planned.
- In response to the results of an internal review, ACOA has recently made changes to better align the IC sub-program with client needs and evolving federal government priorities. These changes may impact demand for funding, types of clients and projects, results and efficiency.
- The IC sub-program tends to complement rather than duplicate innovation and commercialization programming delivered by other organizations as well as other ACOA sub- programs. The IC sub-program is differentiated from other similar programs in its focus on commercialization, level of financial assistance provided, support for riskier projects and clients, and regional delivery model. To minimize the potential for overlap and leverage support for clients, ACOA plays a key leadership role in the region and coordinates activities with other programs. The IC sub- program also works closely with other ACOA programming in supporting clients along a continuum of development.
- The IC sub-program is consistent with the strategic priorities, roles and responsibilities of the federal government and ACOA as well as strategies and approaches implemented by federal organizations in other regions of Canada and other jurisdictions. The IC sub-program aligns well with federal government priorities, particularly as outlined in the Science and Technology Strategy 2007, the Budget Plan 2014, speeches from the Throne and the 2011 Innovation Canada: A Call to Action (commonly known as the Jenkins Report), as well as with the strategic priorities, roles and responsibilities of ACOA. The focus of ACOA’s IC sub-program is consistent with approaches implemented by other federal government organizations (e.g. the high priority placed on innovation and commercialization, an emphasis on market “pull” innovation policy, and use of a broad definition of innovation).
Performance - effectiveness
- The IC sub-program has resulted in extensive direct and leveraged investments in innovation and commercialization projects that, in turn, have generated a wide range of impacts in the immediate and intermediate term. Over the longer-term, the sub-program has expanded innovation and commercialization capacity and contributed to the growth and competitiveness of SMEs in Atlantic Canada. The $80 million in project funding approved annually by ACOA was leveraged with $1.32 in funding from other sources for every dollar contributed by ACOA and stimulated approximately $131 million in applied R&D expenditures, $57 million in commercialization expenditures, project employment for over 1,400 people, the development of 547 partnerships and 405 new technologies, products or processes, the adaptation of 370 technologies, and the commercialization of 521 technologies, products or processes that, as of the latest fiscal year, were generating $74 million in direct and spinoff revenues and 958 jobs. These results are understated in that the impacts will continue to grow as more projects reach the commercialization stage and revenues and employment from commercialization increase over time. The projects have also led to new patents and technology disclosures, further investment in R&D, cost savings and the attraction of new investment. Over the longer term, ACOA funding has supported the establishment and further development of various industrial clusters in Atlantic Canada, including aerospace and defence, life sciences and biotechnology, ocean technologies, information and communications technology, and energy (particularly renewable energy) sectors. The economic analysis, using the Conference Board of Canada’s (CBoC) econometric model, estimates that the IC sub-program generated an increase in gross domestic product (GDP) of $5.49 for every project dollar expended by ACOA over the 2007-2008 to 2012-2013 period.
- The progress made to date has been influenced by the characteristics of the projects supported, the capabilities of client organizations, the design and delivery of the programming and economic conditions. The effective design and delivery of the sub-program has contributed to its success. Impacts vary across projects depending, in part, on focus (e.g. projects focused on commercialization require less time to generate economic impacts), the elapsed time since the project was completed (e.g. impacts increase over time), sector (e.g. certain sectors have shorter commercialization time frames than others), and the client’s technical and business skills. Factors that, at times, constrained project success include delays in project approval processes (i.e. all phases of application processing) or implementation and challenges in the economic environment. Some of the key lessons learned include the importance of being responsive to business needs in program delivery, supporting capacity development, offering a range of assistance to clients that can be tailored to their specific needs, and leveraging resources.
- While ACOA staff have access to project monitoring and performance reporting information, there are specific challenges that need to be addressed. Data is used primarily for monitoring and information gathering, managing the portfolio, adjusting programming and Agency reporting. Challenges associated with performance measurement include difficulties in rolling up outcomes, the time required to see impacts, inherent difficulties in measuring innovation and commercialization, and outcome information not being entered into the system.
Performance – efficiency and economy
- The existing governance structure, delivery model and funding mechanisms appear to support the efficient and economical delivery of the programming (O&M costs average about 8% of total program expenditures). Various factors contribute to the IC sub-program efficiency, including the regional governance structure (e.g. allows regions to tailor strategies and focus on needs), the client-centric delivery approach (e.g. offering a range of support mechanisms), the optimized use of resources (e.g. leveraging and repayability), and the collection and use of high-level performance information (e.g. using dashboards) to monitor portfolio performance and make investment decisions. Further improvements are expected as a result of recent changes to the programming. There may be opportunities to further improve efficiency and economy by sharing tools, processes, promising practices and lessons learned across regions.
Recommendations
The major recommendations arising from the evaluation of the Innovation and Commercialization sub-program are as follows:
- Carefully monitor the recent programming changes to assess the impact and determine the need for further changes or adjustments.
- Based on the results of its recent review, the Agency renewed its commitment to IC and is strengthening its programming by moving to a continuous intake system and lowering the minimum funding threshold to better enable the AIF to meet the needs of business as well as by launching a new BDP commercialization funding element. Some changes have also been made to the delivery structure. With these changes, the programming is expected to better operate at the speed of business and place a greater emphasis on commercialization (without excluding targeted investments in R&D capacity). The results should be carefully monitored to assess the impact of the changes on demand for funding, characteristics of clients and projects supported, approval times, risk, resulting impacts, and ongoing communication and sharing of information, both internally and externally, regarding client needs, recent developments, lessons learned and promising practices.
- Revise the Performance Measurement Strategy (PMS) for the IC sub-program to reflect the recent program changes and improve the utility of the system for both performance reporting and management planning by integrating outcome data into monitoring systems.
- Program logic models are being revised and the results will need to be reflected in a revised PMS. Difficulties in meaningfully rolling up project outcomes have meant that the impact information reported by clients has not been used extensively for either performance reporting or management planning. Limited use of the system has meant that a low priority is often placed on inputting impact or outcome information reported by clients into QAccess. To address these issues, ACOA should consider promising practices such as employing a mix of standard and unique indicators and incorporating a more client-focused approach rather than a more narrow project-focused approach to performance measurement.
Management has agreed with this evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the above recommendations, can be found in Appendix A.
1.0 Introduction
The Atlantic Canada Opportunities Agency (ACOA) promotes economic growth in Atlantic Canada by helping businesses become more competitive, innovative and productive, by working with diverse communities to develop and diversify local economies, and by championing the strengths of Atlantic Canada. ACOA’s mandate is “to enhance the growth of earned income and employment opportunities in Atlantic Canada.”
ACOA’s program alignment architecture (PAA) consists of four programs: Enterprise Development; Community Development; Policy, Advocacy and Coordination; and Internal Services. Enterprise Development consists of three sub-programs: Innovation and Commercialization; Productivity and Growth; and International Business Development. The IC sub-program is delivered primarily through two G&C programs: the Atlantic Innovation Fund (AIF) and the Business Development Program (BDP). In 2012-2013, spending under the IC sub-program accounted for $89.7 million of the $179.9 million dedicated to Enterprise Development.[i] The budget, sub-programs and performance indicators for each program as well as for ACOA overall are summarized in Appendix B.
1.1 Evaluation overview
The evaluation of the IC sub-program is expected to contribute to learning, future decision making and accountability (meeting the Treasury Board Policy on Evaluation requirement that all direct program spending be evaluated every five years). The evaluation covers the areas of program relevance (i.e. continued need for the program, alignment with government priorities, and alignment with federal roles and responsibilities) and performance (i.e. effectiveness, efficiency and economy). The scope of this evaluation includes IC activities approved between 2005-2006 and 2012-2013. This scope overlaps the scope of the previous Innovation evaluation (2010) to enable the evaluation to capture data on the longer-term results and impacts of the sub-program. The issues and questions addressed in this evaluation are provided in Appendix C.
Section 2 describes the evaluation methodology, limitations and mitigation strategies, section 3 provides an overview of the IC sub-program, sections 4, 5 and 6 summarize the major findings related to relevance, effectiveness, efficiency and economy, and section 7 presents the major conclusions and recommendations.
1.2 Evaluation approach and methodology
This evaluation used a hybrid team approach (internal evaluators and external consultants) in implementing a mixed method research design involving multiple lines of evidence. The evaluation complemented a formal internal review of innovation and commercialization activities recently undertaken by ACOA, utilizing some of the information collected and focusing research primarily on issues not covered as part of the review. The major lines of evidence for the evaluation include:
- Document and Literature Review: A comprehensive review, focused primarily on issues related to relevance, was undertaken of internal and external documents related to the sub-program, funding programs, innovation and commercialization, federal policies and strategies, and previous evaluations. In addition, a review of similar activities in other jurisdictions was undertaken to identify alternative modes of delivery, possible lessons learned and best practices.
- Review of Project and Operational Data: QAccess data were used to develop a detailed statistical profile of funded projects, client organizations, partnerships and the interrelationship between the three Enterprise Development sub-programs. In addition, operational data regarding resource allocations was reviewed and used in assessing efficiency and economy.
- Regional Profiles: The results of the document and literature review, project data and operational data analysis were used in developing regional delivery profiles that describe how the sub-program is structured and delivered in each of the regional offices.
- Client Survey: A web-based and telephone survey was undertaken, targeting 530 of the 762 AIF and BDP clients funded through the IC sub-program between 2005-2006 and 2012-2013. A maximum of 489 companies were contacted by e-mail or telephone. Each client was asked about one project, selected by the ACOA Evaluation Team on the basis of materiality (i.e. funding provided) and year of funding, with preference given to projects undertaken in the earlier years covered by the evaluation. Of these 489 clients, 277 responded, which represents 52% of the clients in the sample population, 24% of the projects in the total population of IC projects, and 36% of the unique clients funded from 2005-2006 to 2012-2013. At a confidence level of 95%, the sample of 277 respondents achieves a margin of error of about ±4.0%.
Table 1: Response Rate, by Program and Type of Client
Program |
Type |
Surveyed |
Project Sample |
Response |
---|---|---|---|---|
BDP |
Commercial |
202 |
396 |
51% |
Non-commercial |
28 |
54 |
52% |
|
Sub-total |
230 |
450 |
51% |
|
AIF |
Commercial |
24 |
49 |
49% |
Non-commercial |
23 |
31 |
74% |
|
Sub-total |
47 |
80 |
59% |
|
Total |
277 |
530 |
52% |
Source: ACOA QAccess project management system, data extracted September 2013.
- Case Studies: Sixteen case studies, involving 52 projects (18 AIF- and 34 BDP-funded projects), were selected for in-depth review based on criteria such as region, sector, client characteristics and availability of performance data. The case studies illustrate the nature of the impacts, the lessons learned, contributing and constraining factors, and the interrelationship between projects assisted by ACOA. The case studies involved a review of documents and data (e.g. proposals, tracking forms, progress reports, final reports and publicly available information) and interviews with 53 representatives from proponent organizations, ACOA, other sources of project funding, and others involved in or impacted by the project.
- Key Informant Interviews: Semi-structured interviews were conducted with 60 key informants, including 27 representatives of ACOA (as part of 16 group interviews), 17 representatives of other government departments and partners, 12 stakeholders and experts, and 4 representatives of similar programs delivered by other regional development agencies (RDAs) of the Government of Canada. The primary focus was to obtain input on the need for the programming, its relationship to other programs, factors that contribute to and constrain the achievement of the intended outcomes, lessons learned, best practices and opportunities for improvement.
- Economic Impact Analysis: The results of the client survey and project data were analyzed to provide input into an economic impact analysis that uses the Conference Board of Canada’s dynamic econometric model of the four Atlantic provinces. The analysis estimated the direct, indirect and induced economic impacts of IC sub-program activities.
1.3 Evaluation challenges and mitigation measures
The major challenges associated with the evaluation and the strategies employed to mitigate them are described below.
- Not enough time has elapsed for the impacts of the projects to be fully realized. Only 70% of the 277 projects surveyed were completed at the time of evaluation. Projects that have been completed (particularly certain types of projects such as AIF projects or projects in the health sector) may require additional time post-completion to realize their full impacts. To mitigate this challenge, the selection of projects for the client survey and case studies placed a higher priority on those undertaken in the earlier years covered by the evaluation. In addition, the results of the survey were analyzed to illustrate the growth in impacts over time. Given the sample size, the estimates are considered reliable, although the reported results are understated as impacts are expected to continue to grow.
- The potential for non-response error: Factors that contributed to the potential for non-response error included difficulties in reaching some respondents, the completeness and accuracy of some contact information, and the length of time since the projects were completed (which may impact the availability of clients as well as their willingness and ability to respond). To mitigate these potential effects, we followed-up with target respondents multiple times, offered them the option of responding through multiple modes and, where necessary, identified alternative respondents within the same organization to interview or survey. We also confirmed that the characteristics of the survey respondents were broadly representative of the target population.
- Potential for respondent bias: The majority of those surveyed or interviewed were involved with or beneficiaries of the sub-program. To mitigate this, the purpose of the survey or interview, the design and methodology, and strict confidentiality of responses were communicated clearly to participants. In addition, the sample of key informants included federal, provincial and non-profit representatives who are not directly involved in the sub-program, and the survey and interview results were triangulated with data obtained through other lines of evidence.
Given these mitigation measures, the limitations of the study were adequately addressed and the results of the IC sub-program evaluation are deemed to be reliable and valid.
2.0 Profile of the innovation and commercialization sub-program
2.1 Sub-program profile
Projects coded to the IC sub-program are funded through the AIF and the BDP. Launched in 2001, the AIF aims to strengthen the region’s level of activity in and capacity for innovation, R&D, and commercialization as well as to maximize benefits from national R&D programs.
Launched in 1995, the BDP has a very broad scope, encompassing a variety of eligible recipients and activities. The objectives of the BDP are to improve the growth and competitiveness of Atlantic SMEs, provide for dynamic and sustainable communities in Atlantic Canada, and provide for policies and programs that strengthen the Atlantic economy.
The AIF and BDP target both commercial and non-commercial entities (including research institutions, universities, and other not-for-profit organizations such as industry associations). Indirect stakeholders of the IC sub-program include industry groups and associations, federal science-based departments, national R&D programs and provincial governments.
2.2 Sub-program logic model
As indicated in the logic model in Appendix D, the activities and outputs of the IC sub-program consist primarily of financial assistance for innovation and commercialization projects. In addition, ACOA staff provide in-kind assistance to clients in the form of help in navigating ACOA programming, referrals to other programs, and one-on-one assistance with business planning and strategies.
In the immediate term, the financial assistance is intended to increase applied R&D activity, capacity for applied R&D, adaptation of technology, and capacity for commercialization; to establish networks, partnerships, and alliances; and to attract, develop and retain R&D staff. In the intermediate term, it is expected to result in new or improved technologies, products, processes and/or services; benefits from increased applied research capacity; further investment in R&D; and commercialization of research. In the long term, the sub-program is intended to strengthen the capacity for innovation and commercialization in Atlantic Canada, which will contribute to the intended outcome for the Enterprise Development Program (“improved growth and competitiveness of Atlantic SMEs”) and the strategic outcome for the Agency overall (“a competitive Atlantic Canadian economy”).
2.3 Sub-program expenditures
Grants and contributions
From 2005-2006 to 2012-2013, $653 million in funding was approved, involving 1,131 projects and 762 unique clients. Of this funding, $414 million (64%) was approved through the AIF and $238 million (36%) was approved through the BDP. AIF funding for commercial clients consists of interest-free, unsecured, repayable or conditionally repayable contributions, while the funding for non-commercial clients consists of non-repayable contributions. AIF funding per project ranged from about $300,000 to $10 million in value, averaging about $2.3 million. BDP assistance for commercial clients consists of repayable or conditionally repayable interest-free, unsecured contributions, while assistance for non-commercial projects is in the form of non-repayable contributions or grants. BDP funding per project ranged from nearly $5,000 to $5 million in value, averaging $255,000.
Table 2: Profile of ACOA Assistance, by Program and Type of Client, 2005-2006 to 2012-2013
Program/Client |
Approved Projects |
ACOA Assistance ($M) |
Percentage of Projects That Are: |
|||||
---|---|---|---|---|---|---|---|---|
Repayable |
Conditional |
Non-Repayable |
Grant |
Total |
||||
BDP |
||||||||
Commercial |
790 |
$198.0 |
76% |
22% |
2% |
-- |
100% |
|
Non-comm. |
162 |
$40.1 |
0% |
0% |
90% |
10% |
100% |
|
Total BDP |
952 |
$238.1 |
63% |
19% |
16% |
2% |
100% |
|
AIF |
||||||||
Commercial |
90 |
$209.5 |
3% |
96% |
1% |
0% |
100% |
|
Non-comm. |
89 |
$204.9 |
0% |
0% |
100% |
0% |
100% |
|
Total AIF |
179 |
$414.4 |
2% |
48% |
50% |
0% |
100% |
|
Total |
1,131 |
$652.5 |
53% |
23% |
23% |
1% |
100% |
Source: ACOA QAccess project management system, data extracted September 2013.
The budget allocation for the BDP is guided by regional allocations. Adjustments may be made mid-year, taking into account spending to date and expected expenditures, including any special initiatives.
The AIF does not operate with a regional allocation and investments are based on a competitive process. The AIF was originally approved by the Treasury Board as a $300 million, five-year initiative that was administered in two large rounds (approximately $150 million per round). The second envelope (2006-2009) was administered in four smaller rounds (approximately $60 million per round). In its 2010 Budget, the Government of Canada made the AIF a permanent, ongoing program.
In the Economic Action Plan 2014, the Government reaffirmed its commitment to increasing business innovation, productivity and competitiveness in Atlantic Canada. As part of this commitment, ACOA will contribute $450 million over the next five years to innovation projects through the AIF and BDP[ii]
Details regarding the types of projects supported by the IC sub-program are available in section 5.1.
Operations and maintenance expenditures
Operations and maintenance expenditures related to the IC programming totalled $56 million from 2005-2006 to 2012-2013, including $11 million in operating expenses and $45 million in salaries as shown below.
Table 3: Operations and maintenance expenditures, 2005-2006 to 2012- 2013
Fiscal Year |
Operating Expenditures |
Salary Expenditures |
Total |
---|---|---|---|
2005-2006 |
$1,645,672 |
$4,820,876 |
$6,466,548 |
2006-2007 |
$1,411,099 |
$5,351,742 |
$6,762,841 |
2007-2008 |
$1,745,916 |
$5,380,965 |
$7,126,881 |
2008-2009 |
$1,600,930 |
$5,373,912 |
$6,974,842 |
2009-2010 |
$1,605,532 |
$6,677,926 |
$8,283,458 |
2010-2011 |
$1,558,766 |
$6,018,032 |
$7,576,798 |
2011-2012 |
$980,741 |
$5,302,968 |
$6,283,709 |
2012-2013 |
$722,901 |
$6,049,787 |
$6,772,688 |
Grand Total |
$11,271,557 |
$44,976,208 |
$56,247,765 |
Source: QAccess ACOA GX financial management system, data extracted May 2014.
As shown in the table below, from 2007-2008 to 2012-2013, full-time equivalents (FTEs) estimated to be dedicated to IC programming ranged from 66 to 80, representing about 30% of all FTEs dedicated to Enterprise Development (ED) and about 10% of total ACOA FTEs.[iii] FTEs reflect an estimated proportion of staff time dedicated to IC programming by program officers/economic development officers/policy analysts, payment officers, managers, directors, administrative staff and upper-level management. Delivery structures and types of staff dedicated to IC programming vary by region.
Table 4: FTEs dedicated to innovation and commercialization, 2007-2008 to 2012-2013
Fiscal Year |
IC FTEs |
Total ED FTEs |
Total ACOA FTEs |
---|---|---|---|
2007-2008 |
69.4 |
441 |
726 |
2008-2009 |
66.2 |
424 |
723 |
2009-2010 |
79.9 |
212 |
718 |
2010-2011 |
74.3 |
215 |
714 |
2011-2012 |
70.3 |
203 |
683 |
2012-2013 |
69.5 |
181 |
588 |
Sources: IC FTE data extracted from GX data. Total ACOA FTEs data extracted from Departmental Performance Reports for fiscal years 2007-2008 to 2012-2013.
2.4 Governance and management
Organizational structure
IC sub-program activities are overseen by the Director General (DG) of Enterprise Development located in ACOA’s head office, together with regional directors of Enterprise Development. The DG Enterprise Development reports directly to the Vice-President, Policy and Programs, while the regional directors of Enterprise Development report to the DG of Operations within their respective region. Major programming decisions are made by ACOA’s Executive Committee, which includes ACOA regional vice-presidents and related senior executives as identified by the President to ensure the appropriate representation of expertise and experience.
ACOA’s Head Office is located in Moncton, N.B. Regional offices are located in Halifax, N.S.; Charlottetown, P.E.I.; St. John’s, N.L.; and Fredericton, N.B. Some IC programming had also been delivered through the former Enterprise Cape Breton Corporation, whose economic and community development activities, including the associated budget, were recently transitioned over to ACOA. Each regional office is headed by a vice-president and is responsible for program delivery within its respective province and for supporting Head Office in the development of program policies. Some regions have satellite offices that support the delivery of ACOA programming (e.g. eight in New Brunswick, six in Newfoundland and Labrador, and nine in Nova Scotia).[iv]
Delivery processes
Prior to recent changes[v]AIF projects were approved through an annual competitive process. While project proposals were normally accepted through a request for letters of intent and project proposal process, projects could be accepted on an exceptional basis under the strategic initiatives element of the program, which involved a separate application-based process. The AIF is coordinated and managed through the AIF Secretariat established at Head Office. The Secretariat’s main objective is to guide managers in delivering the AIF program and to coordinate the intake of project proposals. The AIF Secretariat coordinates the scientific review process, specifically the involvement of other federal departments and agencies (e.g. National Research Council Canada). For additional expertise assistance, account managers draw from federal partners or from external consultants. Sector Teams provide advice and perform a challenge function on project proposals. The AIF Secretariat also liaises with the AIF Advisory Board and ensures that they have sufficient information to perform their duties. The AIF Advisory Board is an independent body established to evaluate and recommend projects for funding to the minister and president.
In contrast, the BDP follows a continuous, demand-driven intake process with no specific funding decision rounds. Authority is delegated to various levels depending on the regional office and type of project, and is also established as per the Agency’s financial signing authority delegation instrument. Across the regions, the level of signing authority for a BDP contribution ranges from no signing authority to up to $100,000 for account managers, up to $300,000 for managers, up to $500,000 for directors, director generals of operations, and regional vice-presidents. Above that level, the president can approve contributions up to $ 10 million, however direction from the Minister is sought for all contributions between $1 million up to $10 million.
The regions also differ in their approach to project development. Some regions have committees to review projects (e.g. New Brunswick has a Projects Under Development Committee that meets monthly) and some regions have teams that seek out projects in specific sectors (e.g. Newfoundland and Labrador has Opportunities-Based Teams that focus on developing specific economic opportunities for industries, regions and businesses such as supplier development to the oil and gas industry), and some regions (e.g. Nova Scotia) work closely with partners and the private sector to develop clients in high-growth sectors.
3.0 Findings: relevance
This section summarizes findings with respect to need for the sub-program, relationship with other programming, and alignment with ACOA and federal government priorities, roles and responsibilities.
3.1 Continued need for the programming
The results of the evaluation demonstrate a strong, continued need for the IC sub-program given the importance of innovation and commercialization to regional competitiveness, the need to build on the progress achieved in recent years, and the fit between the IC sub- program and the needs of clients.
Continued need
Innovation and commercialization are major drivers of business productivity and the economic competiveness of a region.
According to the literature, there is a strong link between innovation, productivity and, ultimately, the economic growth and competitiveness of a region. The Organisation for Economic Co- operation and Development (OECD) Oslo Manual states: “It has been long understood that the generation, exploitation and diffusion of knowledge are fundamental to economic growth, development and the well-being of nations.” The Manual defines innovation as “the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new organizational method in business practices, workplace organization or external relations.”[vi]
Innovation leads to enhanced growth and competitiveness by making businesses more productive. The World Economic Forum Global Competitiveness Report defines competitiveness as “the set of institutions, policies and factors that determine the level of productivity of a country.” The report explains that there have been many productivity gains throughout history attributable to innovation and that these gains require an environment conducive to innovative activity supported by both the public and private sectors (e.g. sufficient investment in R&D, particularly by the private sector, the presence of high quality scientific research institutions, extensive collaboration between universities and industry, the protection of intellectual property, high levels of competition, and access to venture capital and financing). The report further states that investment in these activities is critical to sustainable growth.[vii]
Canadian policy studies have also identified the link between innovation, productivity, competitiveness and growth, particularly with respect to the commercialization of innovations. Innovation Canada: A Call to Action explains that productivity growth drives increases in average per capita income and business competitiveness, and that productivity growth is primarily the result of innovation. The report emphasizes that innovation diffusion and adaptation are critical to productivity growth [viii],which is achieved through successful commercialization of innovative technologies, products, processes or services.
Over the past decade, considerable progress has been made in strengthening the capacity for innovation and commercialization and increasing investment in R&D in Atlantic Canada. Some areas of progress highlighted in recent reports include:
- Increased recognition of the importance of research and innovation policies and programs in promoting economic prosperity, as reflected in new programming and plans (e.g. key initiative of the Council of Atlantic Premiers) and greater interprovincial and federal- provincial collaboration.
- Further development of research capacity. Sixteen universities and five college systems perform R&D, train students and researchers, and create spinoff businesses. Eight Atlantic Canada universities were ranked among Canada’s 50 top research universities in 2011 in sponsored research income.[ix] There is a wide range of public and non- governmental research institutes.
- Increased collaboration between academia/research institutions and the private sector, catalyzed in part through an increased emphasis on collaboration in funding provided under various government programs, including ACOA’s. The establishment of Springboard Atlantic (a technology transfer network of 13 universities and five colleges that supports commercialization of university research in Atlantic Canada) in 2004 has strengthened university-industry liaison and technology transfer capacity, among other areas.
- Further development of other resources, networks and ecosystems that facilitate commercialization by providing financing, training and mentoring. Examples include the First Angel Network Association, which delivers education and networking opportunities related to investment readiness and commercialization, Learnsphere, which focuses on skills development, and industry associations, which facilitate partnerships.
- Increased investment in R&D. According to Statistics Canada data, business enterprise R&D (BERD) spending grew at a higher rate in Atlantic Canada (4.7%) compared to the rest of the country (1.1%) from 2001 to 2011.[x]
However, the region continues to lag other parts of Canada and other countries on key innovation and commercialization indicators and faces significant barriers, highlighting the continued need for innovation and commercialization programming. Atlantic Canada continues to lag behind Canada on key innovation measures and productivity, which hinders the ability of its firms to grow and compete internationally. Relative to other Canadian provinces, the region experiences:
- Low levels of BERD and R&D performed by the private sector. Although 7% of Canada’s population resided in Atlantic Canada in 2011, the region accounted for less than 2% of national BERD. Twenty percent of the R&D activities in Atlantic Canada are performed by the business sector, as compared to 60% nationally. There is empirical evidence that R&D performed by business, rather than by universities and governments, contributes most directly to productivity growth.
- Lower productivity and corresponding level of investment in machinery and equipment. Atlantic Canada’s labour productivity in 2011 was equal to 84.6% of the Canadian average.
- More limited access to angel and venture capital financing. Though access to growth capital has increased in recent years (from 2% in 2008), Atlantic Canada received only 4.6% of the venture capital invested in Canada in 2011.
- Lower rate of intellectual property creation. Only 2% of patents filed and granted in Canada are from Atlantic Canada.[xi]
- More limited access to federal research grant funding. Even though the level of federal research grant funding awarded to Atlantic Canada institutions has increased substantially over the past 10 years, the proportion of funding awarded has not increased.[xii] Most of the increase in funding has been awarded to a few larger universities (e.g. Dalhousie, Memorial and New Brunswick), with smaller institutions continuing to face challenges competing for funds due to limited capacity.
Furthermore, Canada lags other countries on similar key innovation measures such as BERD as a percentage of GDP (1% versus 1.6% OECD average in 2011), business-sector labour productivity (75% of the level in the United States), collaboration between business and post-secondary sectors, and doctoral graduates in science and engineering (ranking 21st among OECD countries in percentage of new degrees awarded in science and engineering). According to the CBoC, Canada ranks 13th among 16 peer countries in terms of innovation, lagging particularly in terms of commercialization.
Significant barriers that contribute to the innovation and productivity gaps in the Atlantic region include a historical focus on primary resource industries, a small (regional) domestic market with a highly rural population, significant interprovincial and regional disparities, an aging population, a high level of outmigration, limited success in attracting and retaining immigrants and skilled workers (both technical and managerial), and a risk-averse business culture. Working in association with ACOA Enterprise Development and Community Development sub-programs, the IC sub-program targets these barriers by making strategic investments in projects that expand R&D capacity, build linkages between universities/research institutions and businesses, support the commercialization of research activities and encourage Atlantic SMEs to engage in innovation activities by sharing the financial risk and leveraging other resources from public- and private-sector financial partners.
Responsiveness to needs
IC sub-program funding facilitated the implementation of projects that met the needs of clients and that would otherwise not have proceeded as planned.
Most clients surveyed indicated that the assistance provided by ACOA met their needs, and most case study proponents were satisfied with the assistance received. Only a few of the clients surveyed indicated that ACOA did not meet their needs, noting the following: more funding was needed; greater flexibility was needed with respect to the completion date or how the funding was used; the funding should have been conditionally repayable or non- repayable; or there were delays in processing their application or claims.
ACOA support is seen as critical to project implementation, as only 4% of clients surveyed (0% of AIF and 5% of BDP clients) would have proceeded with the project as planned in the absence of ACOA support. Projects would most likely have been reduced in scope (39%), cancelled (35%), delayed (25%) and/or implemented over a longer period of time (24%). On average, clients estimated that there was a 35% likelihood that the projects would have proceeded in some form without ACOA support. In the absence of ACOA support, 18 percent of surveyed clients indicated that they would have approached another organization or program for assistance, such as the National Research Council, the provincial government, other federal government departments, private-sector investors and financial institutions.
According to the 2014 Innovation evaluation, the percentage of clients who indicated that the project would have been cancelled without ACOA funding was somewhat higher for AIF projects relative to the results reported in the 2010 evaluation (51% would not have proceeded, compared to 46% reported in 2010) and lower for the BDP projects (31% compared to the 46% reported in 2010).
To ensure continued relevance of the IC sub-program, ACOA has monitored the changing innovation and commercialization landscape and very recently adjusted its programming to better meet client needs and reflect evolving federal government priorities.
The results of the document and literature review illustrate that the IC sub-program is responsive to needs in that it strengthens factors that drive innovation and commercialization performance. The Agency commits significant funding to promote innovation and commercialization activities and places an emphasis on the development of SMEs and clusters in key strategic sectors. It facilitates responsiveness through its regional design and delivery model, which enables support to be tailored and adjusted over time to meet existing and emerging needs. Account managers play a critical front-line role in linking ACOA programs with client needs.
A majority of ACOA representatives believe that the Agency has access to the information it needs to adjust its programming and strategies in response to changes in the innovation and commercialization landscape. For example, the Agency recently conducted a formal internal review of innovation and commercialization activities. The review involved extensive consultations with key industry and institutional stakeholders across Atlantic Canada and resulted in adjustments to the IC programming (e.g. an increased focus on commercialization, a more streamlined application process and increased flexibility in types of eligible expenditures).
On an ongoing basis, ACOA tracks changes in the innovation and commercialization landscape on a less formal basis, relying on information obtained from federal partners, provincial partners and other key stakeholders (e.g. industry associations and start-up accelerators), through networks, events and forums (e.g. the Atlantic Venture Forum), clients, program staff, program evaluations and reviews, Statistics Canada and provincial government sources, and research reports. Some ACOA representatives noted that it is difficult to keep up with the pace of change in the innovation and commercialization landscape. Information gaps were identified with respect to broader client or systemic issues (e.g. access to angel investment, venture capital and other support services), data on specific industrial clusters, and the needs of specific areas or segments (e.g. the needs of rural areas).
3.2 Relationship to other programming
The IC sub-program is differentiated from other programs that promote innovation and commercialization in Atlantic Canada by its focus on commercialization, the level of financial assistance provided, its support for riskier projects and clients, and a regional delivery model.
A variety of programs delivered by the federal government, provincial governments and others promote innovation and commercialization in Atlantic Canada. While statistics are not available on investment in commercialization activities in the region, data are available on R&D investments. Statistics Canada data indicate that, in 2011, approximately $1.16 billion was invested in R&D in Atlantic Canada, of which the leading sources were the higher education sector ($429 million), the federal government ($336 million), the business sector ($287 million), the provincial governments ($50 million) and others ($60 million).[xiii]
The following figure draws from data found in Appendix E to provide an overview of leading sources of funding for innovation and commercialization in Atlantic Canada, indicating the average value of G&Cs awarded in Atlantic Canada (based on the most recent year’s data) and the relative position of the program along an innovation continuum from discovery to commercialization. As indicated, ACOA is the largest single source of program funding in the region. The Business Development Bank of Canada does not provide G&Cs but is an important source of funding for commercialization through its loan programs.
Figure 1: Overview of Leading Programs in Atlantic Canada [xiv]
The ACOA IC sub-program is viewed by most key informants and case study proponents as complementing rather than duplicating other programs and sources of financing, noting that ACOA occupies a niche that is not currently filled by other programs. In comparison to other programming, the ACOA IC sub- program differs in the following ways:
- Relative emphasis on commercialization. Both key informants and case study representatives noted that the focus of ACOA’s programming has been shifting away from R&D capacity building toward a greater emphasis on commercialization. Other leading programs (e.g. NSERC and CIHR) tend to focus more heavily on earlier stage research.
- Level of financial assistance available. Key informants, clients and case study representatives noted that ACOA’s IC sub-program complements other programs by offering higher levels of funding for innovation and commercialization projects.
- The types of assistance provided and the nature of projects funded. The IC sub-program offers interest-free, unsecured, repayable, conditionally repayable or non-repayable contributions and supports a wider range of projects, including higher risk projects (e.g. projects in emerging sectors). Few projects would have proceeded as planned in the absence of ACOA funding.
- The regional delivery model. ACOA has a strong staff presence in the regions (60 to 70 FTEs per year dedicated to IC activities delivered by 28 regional and satellite offices). Resources on the ground facilitate greater communication with clients; support the development and processing of applications; and enable the Agency to better understand local needs. Assistance can then be tailored specifically to meet client needs and account for regional context.
Clients confirmed that ACOA’s IC programming offers unique advantages over other existing programs. When clients were asked to rate how important various factors were in their decision to apply to ACOA for support rather than to another organization, the highest rated reasons were the level of funding available, eligibility requirements, timeliness of project approvals, and the definition of eligible expenditures. Responses did not differ significantly by client type (i.e. BDP versus AIF, commercial versus non-commercial).
To minimize the potential for overlap and to leverage support for clients, ACOA plays a key convening role in the region and coordinates activities with other programs.
Some key informants noted a few areas of overlap between the IC sub-program and other programs. It was noted that the AIF overlaps, to some extent, with other federal government funding programs that provide support for applied R&D at universities (e.g. the NSERC Idea to Innovation Grant, which provides funding to college and university faculty members to support R&D projects with recognized commercial potential) and for projects that promote partnerships between universities and SMEs (e.g. Networks of Centres of Excellence). Some overlap was also noted with certain provincial government programs such as the former Department of Innovation Business and Rural Development in Newfoundland [xv] (which offers contribution programs aimed at both businesses and universities) and Innovation PEI (which funds research students).
Most key informants agreed that the areas of overlap are minimal and readily addressed through communication and collaboration between ACOA and its partners. Key informants explained that ACOA coordinates its activities with those of other programs through direct communication with partners regarding support for particular projects, by convening with partners to review programs and coordinate strategies, and by leveraging partners’ scientific and sector expertise. It was noted that ACOA’s involvement in a project can help reduce financing risk and bring additional legitimacy to a project. ACOA’s regional offices also monitor changes in programming support and adapt their focus, as warranted, to fill specific gaps and complement available provincial and federal resources. A few key informants noted that, in some cases, various factors can constrain the level of communication, coordination and collaboration between ACOA and other organizations. These factors include differing organizational objectives and mandates (which can, for example, complicate reporting requirements when funding the same client), the limited involvement of some provincial governments in innovation and commercialization, and difficulties in coordinating activities with programs delivered through departments/agencies outside the region.
The IC sub-program works closely with other ACOA programming in providing direct and indirect support to clients along a continuum of development. Just over one half of clients received funding from another ACOA sub-program during the period covered by the evaluation.
The IC sub-program is viewed by key informants as complementary to other ACOA programming. For example, projects funded under the IC sub-program may result in the development and commercialization of new technologies. Funding provided under the Productivity and Growth (PG) sub-program can support the production and sale of these technologies. Funding under the International Business Development (IBD) sub-program can support the development and expansion of international markets for these technologies. The funding provided to commercial and non-commercial clients works together to support companies along the continuum (e.g. mixing direct assistance to businesses with indirect assistance through non-profit support organizations). ACOA representatives noted that the support provided by the three sub- programs is not necessarily linear but that the programs are complementary because they work toward the same ultimate goals and outcomes of enhanced competitiveness, productivity and growth.
Data from ACOA’s project and client management system (QAccess) indicate that clients are accessing complementary support from ACOA. Of clients funded for one or more IC projects, 48% received funding for IC projects only, 39% received funding for both IC and PG projects, 5% received funding for both IC and IBD projects, and 8% received funding for IC, PG and IBD projects. This complementarity will be further explored in ACOA’s evaluations of IBD and PG, which are currently under way.
3.3 Alignment with federal priorities, roles and responsibilities
Alignment with federal government priorities and ACOA priorities
The IC sub-program is well aligned with federal government priorities as outlined in various strategies, budget plans, and speeches from the Throne as well as with the strategic priorities, roles and responsibilities of ACOA. The primary objectives of the IC sub-program are to support the leveraging of funds toward innovation projects, encourage the establishment of partnerships for technology development and commercialization, and increase revenues resulting from commercialization. These objectives are consistent with the priorities of the federal government, particularly as stated in:
- Canada’s Science and Technology Strategy 2007, [xvi] which states that the Government of Canada will foster a competitive and dynamic business environment and pursue public-private research and commercialization partnerships.
- Budget Plan 2014, which has a specific focus on supporting innovation in Atlantic Canada “to ensure that they are responsive to the needs of business, including small and medium-sized enterprises, and that they effectively support the development and commercialization of new ideas, technologies, products and services that lead to strong economic growth.” [xvii]
- Speeches from the Throne. The 2014 Speech from the Throne emphasizes that “our Government will continue making targeted investments in science and innovation chains from laboratory to market in order to position Canada as a leader in the knowledge economy.” [xviii] Similarly, the 2011 Speech from the Throne highlights that “our Government will continue to make targeted investments to promote and encourage research and development in Canada’s private sector and in our universities, colleges and polytechnics.” [xix}
- Innovation Canada: A Call to Action [xx] emphasizes that priority should be given to fostering collaboration with key actors in the innovation ecosystem: industries, provinces, post-secondary institutions, civil society organizations and relevant investor communities. The scale of collaborative projects should change from investigator-led “idea push” projects to large-scale, industry-facing, demand-driven and outcome-oriented projects.
The objectives are also consistent with ACOA’s “programs and services on initiatives that encourage Atlantic Canadian businesses to become more innovative, productive and competitive in the global marketplace” priority as well as the “competitive Atlantic Canadian Economy” strategic outcome in its 2013-2014 Report on Plans and Priorities, as well as ACOA’s renewed focus on rural SMEs.
ACOA is aware of the priorities and strategies of its partners (federal, provincial, municipal) and fosters alignment where appropriate. Representatives of other government agencies and programs agreed that the sub-program also aligns fairly well with the priorities and strategies of their organizations and those of provincial and regional partners. ACOA’s regional offices work closely with partners to coordinate innovation and commercialization programming. For example, the regional office in Fredericton, N.B., engages the province to examine the roles, responsibilities and structure of economic development partners to increase their impact and efficiency. The Newfoundland and Labrador office places a priority on further developing partnerships/alliances with IC stakeholders. The Prince Edward Island regional office works closely with the provincial government on ED programming and provincial strategies. The provincial government of Nova Scotia actively promotes innovation (e.g. hosting the province’s first Innovation Summit in April 2013).
Alignment with federal government roles and responsibilities
The focus, priorities and roles of ACOA’s IC sub-program are consistent with approaches implemented by federal government organizations in other regions of Canada and internationally.
The IC sub-program is well aligned with federal roles and responsibilities. Although the federal government has long been involved in regional economic development, it was only with the passing of the Constitution Act, 1982 [xxi] that, under section 36, the Government of Canada and the provincial governments committed “to furthering economic development to reduce disparity in opportunities.” The 1988 ACOA Act established ACOA as the primary federal department responsible for economic development efforts in Atlantic Canada [xxii].
ACOA’s roles and responsibilities continue to be relevant even in light of recent developments in the innovation landscape: Jenkins report; the Government of Canada’s intention to release an updated Science, Technology and Innovation Strategy [xxiii]; the transformation of the National Research Council Canada (NRC); and the introduction of the programs such as the Venture Capital Action Plan, the Canada Accelerator and Incubator Program, and the Technology Demonstration Program. [xxiv] Some of the major policy shifts in Canada include the move to market “pull” innovation policy; the adoption of a broader definition of innovation (e.g. to include process and organizational innovation); and an increased focus on SMEs as innovation drivers. Key informants agreed that the sub-program reflects these developments.
The sub-program also aligns well with trends among other federal agencies inside and outside of Canada. Representatives from other Canadian federal RDAs [xxv] described innovation and commercialization as a central or increasingly important priority for their organizations. These agencies are focusing on addressing commercialization gaps, developing strategic partnerships between SMEs and research organizations, and enhancing the productivity of SMEs. Governments in other countries are also placing an increased focus on innovation support: Australia is undergoing a consolidation of innovation programs and is providing targeted support for commercialization (e.g. Commercialization Australia); the United Kingdom supports innovation through public procurement (e.g. Technology Strategy Board), among other strategies; Finland has a complex system of support for innovation, with a particular focus on supporting skills and talent development; and Northern Scotland development organizations such as the Highlands and Islands Enterprise promote innovation by promoting partnerships between businesses and academic institutions, among other mechanisms.
4.0 Findings: performance – effectiveness
This section summarizes the key findings regarding the outputs and outcomes of the IC sub-program, factors contributing to and constraining effectiveness, and performance measurement. Findings are presented at an aggregate level when no significant differences are observed between programs (AIF versus BDP) or client types (commercial versus non-commercial).
4.1 Investment in innovation and commercialization projects
The IC sub-program has facilitated extensive investment in projects, including funding provided by ACOA as well as funding leveraged from other sources.
During the evaluation period, from 2005-2006 to 2012-2013, ACOA approved over $650 million in financial assistance for 1,131 innovation and commercialization projects involving 762 unique clients (an average of 1.5 projects per client). For every dollar of funding provided by ACOA, $1.32 was provided by other sources, including funds contributed by the client organization (reported by 55% of projects surveyed), provincial government (38%), other federal government sources (29%), private-sector partners (26%), private equity (19%), commercial lenders (16%) and universities or research institutions (10%). The leverage rate exceeds the target rate of $1.05 established by ACOA but is lower than the leverage rate reported in 2010 ($1.44).
The results of the client survey, case studies and key informant interviews indicate that the presence of ACOA funding better enables clients to access and use funding from other sources. Clients estimated that, on average, only 34% of the funding from other sources would still have been invested in their project if they had not received assistance from ACOA. Case study representatives emphasized the role of ACOA assistance in fostering the sharing of project risk and better enabling projects to access private-sector financing. Key informants and case study representatives noted that project funding from ACOA and other public-sector partners is often mutually conditional.
The nature of the innovation and commercialization projects and the resulting impacts vary significantly depending upon the program under which they are funded (BDP or AIF) and the type of client (commercial or non-commercial).
The IC sub-program is best viewed as a series of four interrelated program areas (AIF commercial, AIF non-commercial, BDP commercial and BDP non-commercial) that work together to promote innovation and commercialization. While these program areas work jointly toward the intended outcomes of the sub-program, they vary significantly in terms of the types of clients served, projects supported, time frames, and nature of impacts generated.
Appendix G provides a profile of each program area in terms of the client and project characteristics. As indicated, AIF commercial and non-commercial projects each accounted for only 8% of the 1,131 projects approved but 32% and 31% respectively of the $653 million in funding approved by ACOA. In contrast, the BDP commercial clients accounted for 70% of projects but only 30% of ACOA funding, while BDP non-commercial projects accounted for 14% of projects and 6% of funding. Non-commercial projects are most commonly undertaken by universities or colleges, research institutions, industry associations and other non-profits, while commercial projects are, with few exceptions, undertaken by companies or partnerships. The leverage rate is lower for commercial projects than for non-commercial projects, which tend to have greater access to other sources of government funding.
In comparison to BDP projects, AIF projects tend to be much larger in value (average ACOA contribution of $2.3 million compared to $250,000 for BDP projects) and implemented over a longer time (four to five years versus 15 months). In part because of the size of ACOA’s contribution and lower degree of leverage, AIF projects were more likely than BDP projects to be cancelled (as opposed to proceeding as planned or being delayed, extended or reduced in scope) in the absence of ACOA funding.
AIF projects focus primarily on R&D activities (in the client survey, R&D activities accounted for 79% of AIF commercial project expenditures and 88% of AIF non-commercial expenditures). Expenditures associated with BDP projects (both commercial and non- commercial) are more evenly distributed between R&D and commercialization activities. As indicated below, clients receiving AIF funding were much more likely to indicate the primary focus of their project was the development of new technologies, products, services or processes than were BDP clients, whose projects were more likely to focus on later stages in the innovation process (e.g. demonstration, commercialization, marketing, and technology refinement, evaluation and adaptation).
Figure 2: Primary Focus of the Projects Surveyed, by Program Area
Source: Client survey. Recognizing the nature of some projects, the clients surveyed were allowed to select more than one primary area of focus. As a result, the numbers add up to more than 100%.
AIF commercial projects focus on the development of new products, processes and technologies or the refinement of existing ones across a range of sectors such as health, information and communications technology, renewable energy/energy efficiency, and manufacturing. According to QAccess data, expenditures of AIF commercial projects consisted primarily of wages (36%) and machinery and equipment (23%).
AIF non-commercial projects involve broader, systemic or cluster development initiatives (most commonly focused on health, agriculture, aquaculture and food, oil and gas, oceans, and pulp and paper) undertaken by universities or other non-profit organizations. They focus most often on the further development of applied R&D capacity, the implementation of R&D projects to develop new products, processes and technologies that can be commercialized by Atlantic Canadian companies, and the facilitation of technology transfer and commercialization. Leading examples include funding for Springboard Atlantic, which facilitates technology transfer from universities and colleges and promotes commercialization by companies in Atlantic Canada, work by Genome Canada in the early stage development of cod aquaculture, development of a research facility and implementation of a brain research program at Dalhousie, and a human genetics project at Memorial University. According to QAccess data, project expenditures consisted primarily of wages (41%), building structures (14%) and machinery and equipment (10%).
BDP commercial projects assist businesses in a wide variety of ways, such as developing new technologies, products and processes (in-house or through an R&D centre), refining existing technologies, products and processes, technology acquisition, skills development, and marketing. The projects focus on assisting businesses to grow and improve their competitive position (e.g. by achieving cost-savings through the acquisition of advanced technology). According to QAccess data, expenditures consisted primarily of machinery and equipment (47%), building and major renovations (12%), and wages (12%).
BDP non-commercial projects focus primarily on increasing the capacity and skills of participant organizations to undertake applied R&D and to commercialize new technologies, products, services and processes. Similar to AIF non-commercial projects, these projects support increased research capacity (e.g. by supporting infrastructure development), capacity for commercialization and the development of clusters in areas such as agriculture and food, aquaculture, health and medical technology, ocean technology, and bioscience). However, there is greater flexibility under the BDP to support a range of targeted activities (e.g. strategic planning, marketing studies, hiring of experts, staging of events, training and counselling). Leading examples include funding to establish a food product lab at a college, new research facilities at a research institute, a high-performance computing infrastructure for a network involving four universities in the region, a technology demonstration project, and a program that supports research internships involving students and recent graduates. Project expenditures consisted primarily of machinery and equipment (38%), building and major renovations (17%), and wages (16%).
There is some overlap between the AIF and the BDP in terms of clients. During the evaluation period, 46 of the 762 unique clients undertook both AIF and BDP projects under the IC sub-program, of which 26 were businesses [xxvi]. The case studies illustrate that the two programs are often complementary.
Research and infrastructure developed in earlier AIF projects was utilized or built upon in subsequent projects. In some cases, new technologies or products developed with AIF non- commercial and commercial assistance were later commercialized with BDP assistance. Some commercial clients reported that their project benefited from assistance provided by other projects supported through the IC sub-program (e.g. services provided by Springboard Atlantic or Learnsphere). In cases where a client undertook several projects, the projects often targeted similar outcomes (e.g. improved efficiency and business expansion).
4.2 Achievement of expected outcomes
This section presents the findings regarding impacts generated by projects funded under the IC sub-program in the immediate, intermediate and long term. The findings draw primarily from the client survey, the case studies and data extracted from QAccess.
The reported results are understated in that the impacts will continue to grow over time. Many of the projects had not yet generated impacts because they were still ongoing at the time of the evaluation. (Even though the research targeted older projects and overlapped with the previous evaluation’s scope, only 70% of the 277 projects surveyed were completed.) While some projects are able to report impacts before the project is fully completed, many require time post-completion before impacts such as increased revenues will occur. Furthermore, among projects that have already generated impacts, the effects tend to increase over time. For example, most clients that have already commercialized the results of their projects indicated that associated revenues are increasing over time (e.g. resulting revenues were higher last year than the previous fiscal year; a regression analysis indicates that revenue is positively correlated with length of time from project completion, although elapsed time is only one of the factors that affect resulting revenues).
The impacts of AIF projects are particularly understated because, as previously noted, they are more likely to focus on earlier stage R&D than are BDP projects (and therefore tend to require more time post-completion to generate impacts) and are much longer in duration (an average of four to five years in length versus 15 months for BDP projects). The survey sample was drawn from projects funded from 2005-2006 to 2012-2013 (the average start date was 2009), which means that many of the AIF projects were still under way (only 57% were completed at the time of the survey) or only recently completed.
The impacts of non-commercial projects also tend to be understated somewhat because it is easier to measure and attribute resulting impacts for commercial than non-commercial projects. Attributable impacts of non-commercial projects are often indirect (e.g. accruing to participants in project activities or spinoff companies rather than the client organization). For example, in the absence of a dedicated evaluation of its activities, it is not possible to report the aggregate impacts of projects such as Springboard Atlantic that work with a large number of institutions and/or companies.
The impacts have been extrapolated to the total population of projects funded under the IC sub-program during the eight-year period (2005- 2006 to 2012-2013) by aggregating the reported results, expressing the results on a per-dollar-of- ACOA-funding basis, and weighting the results by the level of ACOA funding provided by program and type of client. For ease of interpretation, the extrapolated results have been reported on a typical year basis by multiplying the impacts per dollar by the average level of funding approved by ACOA annually in each program area. [xxvii] Given the sample size, the estimates are reliable, although, as noted earlier, the reported results are understated in that the impacts will continue to grow over time.
4.2.1 Immediate outcomes
The intended outcomes of the IC sub-program in the short term include increased applied R&D activity; the attraction, development and retention of R&D staff; increased capacity for applied R&D; increased adaptation of technology; the establishment of networks and meaningful partnerships/alliances; and increased capacity (i.e. knowledge, skills, and infrastructure) for commercialization. The table below presents the percentage of surveyed clients that reported each type of impact and uses the data reported to forecast what the aggregate immediate impacts would have been, on an average annual basis, if all the clients had been surveyed about all of the projects undertaken during the period covered by the evaluation.
Table 5: Immediate Impacts of IC Sub-program Projects, Expressed on a Typical Year Basis
Impact |
Percentage of Clients Reporting Impact to Date |
Projected Impact of Projects Approved in a Typical Year |
---|---|---|
Investment in R&D |
76% |
$131 million |
Investment in commercialization |
67% |
$57 million |
Development of new partnerships |
73% |
547 |
New private-sector partnerships |
71% |
471 |
Technologies adapted |
69% |
373 |
Average full-time staff on the project |
95% |
1,035 |
Average part-time staff |
61% |
398 |
Newly hired staff |
79% |
728 (51% of all full-time and part-time project employees) |
New staff involved in R&D activities |
64% |
368 (51% of all newly hired employees) |
New staff who continued to be employed following project completion |
77% |
528 (71% of newly hired employees) |
Note: Average annual ACOA assistance (2005-2006 – 2012-2013) was $82.4 million.
Clients were also asked to rate the progress made by their projects against the intended immediate outcomes. Average ratings vary by outcome and program area, depending on the extent to which particular projects targeted the intended outcome as well as the progress made to date. For example, most projects allowed staff who worked on the project to further develop skills and abilities while fewer targeted improving the organization’s marketing or business management practices. [xxviii]
Significant progress has been made toward the achievement of immediate outcomes, particularly in terms of increasing applied R&D activity and capacity; establishing networks, partnerships and alliances; and attracting, developing and retaining R&D staff. The results are discussed below.
On an annual basis, the IC sub- program has stimulated investment of approximately $131 million in applied R&D, including funding provided by ACOA and funding leveraged from other sources.
As a group, AIF clients reported that 85% of their project expenditures related to R&D, while BDP clients reported that 39% of expenditures related to R&D. The projects invested $1.59 in R&D activity ($1.24 and $2.19, respectively, for AIF commercial and non-commercial projects, and $1.32 and $1.06, respectively, for BDP commercial and non- commercial projects) for every dollar provided by ACOA.
Based on the average annual funding of $82.4 million approved by ACOA, the projects supported by ACOA annually are projected to invest approximately $131 million in applied R&D activity. Of this total, $51 million is invested by commercial clients. The level of investment in business R&D is significant; according to Statistics Canada data, the value of R&D funded by businesses in Atlantic Canada was $287 million in 2011. [xxix]
According to QAccess data, of the 1,131 projects approved by ACOA over the eight-year period of this evaluation, 44% focused specifically on applied R&D activity (i.e. the project was coded to technology development or new product and process development). These projects accounted for 67% of the total costs of the projects approved under the IC sub-program.
Projects approved annually under the IC sub-program have employed over 1,400 people (of whom 51% were newly hired) and facilitated skills development for about 350 students each year.
The development of staff skills and abilities (clients reported an average rating of 4.3 where 5 is “to a great extent”), further development of the skills and abilities of any students who worked on the project (average rating of 4.3), and the increased human resources development capacity of the organization (3.9) were the immediate impacts rated the highest by clients, particularly by non-commercial clients.
Ninety-five per cent of clients reported employing project workers on a full-time basis and 61% reported employing project workers on a part-time basis. The organizations (including those who did not employ project workers) employed an average of 7.3 people full-time and 2.9 people part-time to work on the project, of which 51% were newly hired and 27% were both newly hired and involved in R&D activities. Of the newly hired workers, 71% continued to be employed by the organization (or are expected to be) following completion of the project.
Thirty-five per cent of clients reported that their project involved students. Expressed on an annual basis, these projects have enabled an average of 350 students to develop their skills and abilities (mostly through projects undertaken by non-commercial clients, particularly those funded through the AIF).
In addition to staff and student development, capacity for applied R&D in Atlantic Canada has increased as a result of undertaking most projects in-house and further investing about $38 million in research-related equipment and infrastructure annually.
When asked to rate the impact of the project in increasing the capacity for applied R&D, on a scale of 1 to 5, where 5 is a major impact, clients provided an average rating of 3.5. Commercial and non-commercial AIF clients, which focus more directly on applied R&D capacity building than do BDP clients, provided higher average ratings of 4.0 and 4.3 respectively.
Of the research undertaken by the projects surveyed, 87% was undertaken in-house (including 74% undertaken entirely in-house) while 13% was undertaken entirely through an external organization (e.g. an R&D centre). According to QAccess data, $38 million of the average annual funding approved by ACOA was budgeted for capital investments in equipment and other infrastructure by projects that focused on applied R&D activity, of which 62% was invested in commercial projects and 38% in non-commercial projects.
The IC projects have facilitated the adaptation of approximately 370 technologies annually.
Sixty-nine per cent of clients surveyed reported that their project resulted in the adaptation of one or more technologies, although this was not necessarily the primary focus of the project. According to QAccess data, projects that focused specifically on technology adaptation, acquisition or diffusion accounted for 25% of projects, 13% of ACOA funding approved, and 17% of total project costs over the eight years. Clients provided an average rating of 3.8 with respect to the impact of the project on the adaptation of existing technology for use by their organization or others.
The IC sub-program has contributed to the establishment and expansion of networks, partnerships and alliances, including the development of an average of 547 partnerships and alliances annually, of which about 86% have been between the client and private-sector organizations.
A majority of clients (60%) reported that projects were undertaken in collaboration with other organizations. On average, clients reported an average of 3.3 other participants, 59% of which were private-sector businesses. Projects can result in new partnerships and alliances with project collaborators as well as with others, including spinoff companies, technology recipients, joint venture partners, investors, major buyers and others. For example, in one case study, a biotechnology start-up received assistance under the BDP to hire a consultant to assist in preparing a business plan that was used to attract major industry partners from the United States and Europe. In another case study, a manufacturing business partnered with an Atlantic university, with assistance under the AIF, to develop biodegradable packaging films and antimicrobial materials. The project led to discoveries that could have applications in other products and markets (e.g. surgical or medical equipment).
On average, clients reported that their projects led to the development of 3.7 new partnerships and alliances, 86% of which were with private-sector organizations. Partnerships and alliances among private-sector firms and institutions (e.g. universities, colleges and research institutions) are of particular relevance for AIF projects. According to a review of AIF project data, the proportion of projects involving partnerships between commercial organizations and institutions has increased over time for both projects led by institutions and those led by commercial clients, as shown in Figure 3.
Figure 3: Partnerships Between Institutions and Private- Sector Organizations in AIF Projects, by AIF Round [xxx]
Source: ACOA QAccess project management system, data extracted September 2013.
The IC sub-program has expanded commercialization capacity in Atlantic Canada, although more needs to be done. On an annual basis, it has stimulated investment of approximately $57 million in commercialization activities (including ACOA funding and that leveraged from other sources).
A majority of projects (67% of those surveyed) reported at least some expenditures related to commercialization. BDP clients reported that 46% of their project expenditures related to commercialization, while AIF clients reported that 14% of expenditures related to commercialization. The projects invested $0.69 in commercialization activities ($0.22 and $0.25, respectively, for AIF commercial and non-commercial projects, and $1.52 and $1.06, respectively, for BDP commercial and non- commercial projects) for every dollar provided by ACOA. Based on the average annual ACOA assistance of $82.4 million, IC projects are projected to invest approximately $57 million annually in commercialization.
Clients provided an average rating of 3.7 with respect to the impact of the project in strengthening the capacity of their organization to commercialize new technologies, products, processes or services (e.g. knowledge, skills and infrastructure related to commercialization). They provided a rating of 3.2 with respect to improving marketing efforts, and 3.2 with respect to improving business management practices. Commercial clients improved their commercialization capacity in a variety of ways, such as further developing internal staff capabilities (for example, as part of one case study project, a company retained the services of an executive search firm to recruit a qualified candidate for the new position of chief executive officer to drive the company’s expansion forward; the project resulted in increased employees, sales and equity financing for the company), engaging experts to develop business plans or attract investment, upgrading or expanding production capabilities, and developing marketing plans.
In addition to the support provided directly to commercial clients to facilitate capacity development, AIF and BDP funding has also been provided to non-commercial clients that deliver services related to commercialization. Non-commercial projects helped by facilitating access to services in areas such as technology transfer, angel investments and advisory services. For example, Springboard Atlantic received AIF funding to establish a network of university and community college technology transfer and industry liaison offices in Atlantic Canada involving 17 institutions as well as BDP funding to deliver innovation and commercialization workshops for start-up technology companies.
Though the capacity for commercialization has been further developed, most key informants indicated that further work is required. Both the literature and key informants noted that significant progress has been made over the past 10 years in developing R&D capacity and capabilities. The emphasis is now shifting somewhat from research to development. While ongoing investments will be required to sustain and further develop research capacity, the major challenge is to better facilitate the commercialization of new technologies, products, services and processes.
4.2.2 Intermediate outcomes
The intermediate intended outcomes of the IC sub-program include the development of new or improved technologies, products, processes and services, other benefits arising from the increased capacity for applied R&D (e.g. patents, technology disclosures and further investment in R&D) and commercialization of research results. The table below shows the percentage of clients reporting each type of outcome and uses the data to project the aggregate intermediate outcome on an average annual basis. As indicated, clients were much more likely to report progress in the development and commercialization of new technologies, products, processes and services and further investment in R&D than patents, technology disclosures, licensing agreements or spinoff companies.
Table 6: Intermediate Outcomes of IC Sub-program Projects, Expressed on a Typical Year Basis
Outcome |
Percentage of Clients Reporting the Impact |
Projected Impact of Projects Approved in a Typical Year |
---|---|---|
$82.4 million |
||
Development of new technologies, products, processes and services |
67% |
405 |
Issuance of patents |
18% |
114 |
Issuance of technology disclosures |
16% |
212 |
Further investment in R&D |
58% |
$66 million |
Commercialization of technologies, products, processes and services |
65% |
521 |
Licensing or transfer agreements |
27% |
204 |
Development of spinoff companies |
13% |
41 |
Note: Average annual ACOA assistance (2005-2006 to 2012-2013) is $82.4 million.
Clients were also asked to rate the progress made by their projects against the intended intermediate outcomes. The average ratings were higher for the development of new or improved technologies, products, processes or services (4.2), commercialization (3.4) and further investment in R&D (2.8) than for licensing arrangements or transfer agreements (1.9), new patents filed (1.7), technology disclosures filed (1.6), or the establishment of spinoff companies (1.5), all of which are relevant to only a subset of the clients surveyed. [xxxi]
On an annual basis, the IC sub- program has supported the development of 405 new technologies, products, processes or services.
In the client survey, 68% of clients indicated that the project led to the development of new technologies, products, processes or services (an average of 3.3 per project), most of which were characterized as commercialized (46%) or market ready (32%). The 52 projects included in the case studies reported the development of 58 new products, processes and services as well as five new discoveries. Examples include biodegradable and antimicrobial packaging, an air ventilation system that controls both heat and humidity, a web-based artist management software, DNA tests that can be used to predict the reproductive longevity of livestock, and a vaccine platform that can have multiple applications.
On an annual basis, to date, the projects have resulted in 114 patents, 212 technology disclosures and $66 million in further investment in R&D.
Of the clients surveyed, 59% reported that the project led to further investment in R&D; 18% reported the issuance of one or more patents; and 16% indicated that it led to technology disclosures. The majority of patents (55%) and technology disclosures (66%) are associated with AIF non-commercial projects, while commercial projects (both BDP and AIF) accounted for the majority of investment in R&D (86%). Case study projects reported that 50 new patents were issued across a range of sectors and in multiple countries. It should be noted that patents and technology disclosures indicate that intellectual property has been created; however, they are not necessarily indicative of commercial potential. Further investment in R&D can be associated with further developing the technology that was the focus of the project or to undertake R&D on other related or non-related technologies.
Sixty-five per cent of clients reported progress toward the commercialization of technologies, products, processes or services, of which 68% reported increased revenues in the latest fiscal year.
On an annual basis, the projects have supported the commercialization of 521 technologies, products, processes or services to date. The projects have contributed to commercialization by supporting technology development and by supporting the commercialization of technologies that had already been developed. The majority of the impact (67%) is associated with BDP commercial projects.
Thirteen per cent of clients reported that the results of ACOA funded projects contributed to the establishment of spinoff companies.
On an annual basis, the projects have contributed to the establishment of 41 spinoff companies. The majority of the spinoffs resulted from non-commercial projects, including BDP and AIF projects.
On average, however, clients estimated that there was a 27% likelihood that the companies would have been established even if the project that received funding from ACOA had not been undertaken.
4.2.3 Long-term outcomes
The long-term outcome focuses on the growth and competitiveness of Atlantic SMEs, which can be reflected in indicators such as productivity improvements, revenues, employment and wages. As indicated in the table below, 80% of clients reported outcomes in terms of ongoing employment and wages, slightly over half reported increased revenues from commercialization, 42% reported cost savings or productivity improvements, and 30% reported the attraction of new investment.
Table 7: Long-Term Outcomes of IC Sub-program Projects, Expressed on a Typical Year Basis
Outcome |
Percentage of Clients Reporting the Impact |
Projected Impact of Projects Approved in a Typical Year |
---|---|---|
Cost savings/productivity improvements |
42% |
$26 million |
Attraction of new investment |
30% |
$110 million |
Revenues from commercialization |
53% |
$48 million |
Ongoing employment |
81% |
750 |
Ongoing wages |
81% |
$37 million |
Revenues of spinoff companies |
9% |
$26 million |
Employment of spinoff companies |
11% |
208 |
Note: Average annual ACOA assistance (2005-2006 to 2012-2013) was $82.4 million.
When asked to rate the progress made by their projects against the selected outcomes, the long- term outcome rated highest (3.3) by clients was improvements in competitive position (e.g. market share, revenues or profitability of a business). Lower average ratings were provided with respect to cost savings/productivity improvements (2.5) and further investment in the organization by others (2.0), which is relevant only to some projects. [xxxii]
On a per-year-of-funding basis, clients report that the projects have generated $48 million in direct revenues to date and employment for 750 people. In addition, the spinoff companies have generated a further $26 million in revenues and employment for 208 people.
The clients that were surveyed reported that, in their latest fiscal year, they generated an average of $330,000 in revenues [xxxiii] from the commercialization of new technologies, products, processes or services resulting from the project. The average is up sharply from the prior year (when the clients reported an average of $160,000 in increased revenues), suggesting the revenues generated by the projects are still increasing. Of the aggregate revenues reported in the most recent fiscal year, 54% were generated from exports, 29% were generated from sales within Atlantic Canada, and 17% were generated from sales to other parts of Canada. On a per-year basis, the increased employment reported by the clients is equal to 750 positions, including employment with clients that are already generating revenues as well as those still working toward commercialization. Ninety per cent of the reported increase in revenues and 77% of the increase in employment was associated with BDP commercial projects.
Results from the case study cross analysis also point to the achievement of growth of Atlantic SMEs. In the case studies, clients reported the creation of 142 new jobs, the maintenance of 42 jobs, the expansion of 16 companies and increased revenues of $34 million.
On a per-year-of-funding basis, clients report that the projects generated approximately $26 million in cost savings or productivity improvements and the attraction of $110 million in new investment.
Of the cost saving and productivity improvements, 88% are associated with BDP projects, including 82% that are the result of BDP commercial projects. The savings most commonly resulted from new capital investments or the acquisition of new technologies or processes. According to QAccess data, 64% of the total budget for BDP commercial projects involved capital investments, particularly the purchase of new machinery and equipment and the development of new buildings and structures.
The $110 million in new investment was most commonly associated with the BDP program (57%). Some of the case study clients were particularly successful in attracting new investment. For example, one case study company that received AIF and BDP support was successful in attracting almost $40 million in equity and government assistance. Another company that also received support under the AIF and BDP recently secured a significant multi-million dollar investment from a single investor.
Over three quarters of clients reported that the projects enhanced their organization’s competitive position and nearly half reported broader social benefits.
Of those clients surveyed, 76% reported improvements in their organization’s competitiveness (a rating of 3 or more). These clients indicated that their competitive position has improved through strengthening company capabilities and capacity (23%), developing new markets or expanding existing markets (21%), improving productivity (17%), increasing market share (15%), introducing new products, processes or services (12%), and attracting new investment into the company.
Forty-four per cent of the clients reported the project generated social benefits such as environmental, safety or health benefits. Examples that were highlighted in the case studies include development and use of a vaccine to treat cancer patients in Halifax, development of renewable energy technology and reduction of reliance on plastics (electronic prescription cards).
Over the long term, ACOA’s funding of projects has supported the establishment and further development of a variety of industrial clusters.
While considerable work remains to be done, the results of the evaluation indicate that progress has been made since the 2010 evaluation of innovation programming, which found that clusters in Atlantic Canada were still in the early stages of development. The case studies illustrate how the BDP and AIF have contributed to progress in developing Atlantic clusters in the aerospace and defence, life sciences and biotechnology, ocean technologies, information and communications technology, and energy (particularly renewable energy) sectors. For example, since 2005 ACOA has provided support for seven BDP projects and one AIF project undertaken by PEI BioAlliance that have assisted the cluster in growing from 16 companies employing fewer than 450 employees to 38 companies employing over 1,100 people and generating $124 million in revenues; the cluster has developed an international reputation, which makes it easier to attract highly skilled labour to the region.
In addition, 11 BDP non-commercial clients surveyed reported contributing to cluster development through activities such as attracting partners, increasing international awareness and recognition of a cluster, promoting activities in rural areas, solving a technological or production issue that broadly affects the cluster, expanding facilities that can be used by businesses, and better enabling businesses to expand production, develop value chains and increase trade with other parts of Canada.
The economic analysis, using the Conference Board of Canada’s econometric model, estimates that the IC sub-program generated an increase in gross domestic product (GDP) of $5.49 per dollar of funding provided by ACOA over the 2007-2008 to 2012-2013 period.
The Conference Board of Canada’s (CBoC) econometric models were used to evaluate the economic impacts of IC projects funded under the AIF and the BDP. The analysis incorporates project expenditure data reported in QAccess as well as the results of the client survey regarding direct and spinoff revenues generated from the commercialization of products resulting from IC projects. The data has been adjusted for incrementality (i.e. the likelihood that the project would have proceeded with the same scope, at the same time and in the same location without government assistance) and attribution (i.e. the benefits attributed to ACOA are equal to the ratio of ACOA investment to total government funding).
The analysis focuses on the expenditures of IC projects from 2007-2008 to 2012-2013. During this period, IC project expenditures totalled $1.2 billion, of which $513 million was funded by ACOA. According to the CBoC model, these projects have contributed $2.8 billion to the GDP of Atlantic Canada, which represents an increase in GDP of $5.49 for every project dollar expended by ACOA. The projects have generated 32,000 person years of employment, which represents about one person year of employment for every $16,000 in ACOA funding provided. The impact of the projects will continue to grow as more projects reach the commercialization stage and as revenues and employment from commercialization increase over time.
The reported economic impact of the BDP projects to date is greater than the impact of the AIF projects. AIF projects tend to require more time post-completion to generate impacts because of their longer-term focus, their longer duration (an average of four to five years in length versus 15 months for BDP projects), and were less likely to be completed at the time of the survey (only 57% were complete). According to the CBoC model, BDP projects have generated $1.9 billion in Atlantic Canada GDP and 22,000 person years of employment to date, which represents an increase in GDP of $7.94 for every project dollar expended by ACOA and one person-year of employment for every $10,700 in ACOA funding provided. To date, AIF projects have generated $948 million in Atlantic Canada GDP and 12,000 person years of employment, which represents an increase in GDP of $3.41 for every project dollar expended by ACOA and one person-year of employment for every $27,800 in ACOA funding provided.
4.3 Contributing and constraining factors and lessons learned
The progress made to date toward the expected outcomes has been influenced by the characteristics of the projects supported, the incidence of highly successful projects, economic conditions, the capabilities of companies in the region, the nature of the programming and the use by clients of other resources supported by ACOA.
Major factors that influenced progress toward expected outcomes included the following:
- The project type, focus, timing and sector. The evaluation involved a survey of projects about four years, on average, after they had been funded. As a result, projects positioned later in the innovation spectrum (e.g. focused on technology commercialization or adaptation) tended to generate a greater return to date per dollar invested in terms of cost savings, revenues and employment but have less of an impact in terms of incremental ongoing R&D expenditures. The sector in which the project takes place also affects the nature, timing and scale of impacts (for example, projects in the manufacturing and the information and communications technology sectors tended to report greater impacts to date while projects in the health & biotechnology sectors tend to involve a much longer path to commercialization due to the costs, the complexity of the research and regulatory processes. [xxxiv]
- The incidence and magnitude of highly successful projects. Research, development and commercialization is an inherently risky activity. Government funding aims to reduce or spread this risk and encourage business investment in these activities. Given the risk profile, some projects will fail to meet their objectives (e.g. the R&D may not result in a marketable product, the proponent may be late to market, or the product launch may not be successful) while others may achieve only moderate success. The aggregate impact of any innovation program is largely determined by the presence of a comparatively small percentage of highly successful projects. These successes often require a range of assistance (e.g. R&D, development of production capabilities and international marketing) to realize their full potential. The reported impacts were reviewed to determine how many projects it took to account for at least half of the aggregate impact reported by clients. Three projects accounted for 52% of the reported spinoff companies, four projects accounted for 52% of the investment attracted, 10 projects accounted for 52% of the revenues from commercialization, 13 projects accounted for 51% of further investment in R&D, and 17 projects accounted for 51% of ongoing employment.
- Economic conditions and the competitive environment (e.g. economic outlook, global competition, pace of change, market demand and access to other funding). The economic slowdown, which peaked in 2008, impacted the progress made by reducing the demand for funding, the willingness of businesses to take on the risk associated with new projects, access to funding for development and commercialization, and market demand for products. At the project level, clients emphasized that projects were successful when there was strong market demand, the product reached market ahead of the competition, and the product had clear competitive advantages. Availability of resources and investment also contributed to progress. For example, several key informants pointed to the growth of the oil and gas sector in Newfoundland and Labrador as contributing to increased provincial government fiscal capacity and availability of private-sector support for R&D. Some key informants, clients and case studies also indicated that the collaboration between federal and provincial partners contributed to the success of projects by providing additional resources and support to projects.
- The client organization’s own business practices, knowledge and capabilities (e.g. management skills, reputation, contacts and expertise). A qualified team, management buy-in and drive, good reputation with existing contacts, established governance and management structure, expertise in the field, and knowledge of the sector and market needs, all contributed to the development of high quality new products, technologies, processes or services and success in attracting private-sector partners and investment and in generating revenues. According to some key informants, shortages in management skills, difficulties in attracting skilled workers (given the small population base), a shortage of experts that can facilitate commercialization (e.g. intellectual property lawyers, accountants and experts in business development), a risk-averse business culture and limited capacity for R&D continue to constrain innovation in the region.
- The nature of ACOA’s IC programming. A few clients, case studies and key informants highlighted that ACOA’s ability to be flexible in making amendments, the low-risk, non-dilutive [xxxv] nature of ACOA’s financial support (which better enables clients to leverage investments from other federal programs, provincial programs and private-sector organizations), the wide range of financial and capacity building support provided by ACOA, and the capabilities and support of ACOA staff, all contribute to project success.
- The interrelationship with other investments made by ACOA. In the survey, clients were asked first whether they used various resources supported by ACOA and, if so, about the impact of those resources on the success of their project (on a scale of 1 to 5, where 5 is to a great extent). Apart from other projects undertaken by the client organization, the resources that were used most commonly by the clients included trade shows and missions, the technology transfer services of Springboard Atlantic and incubators. The highest overall average rated services were incubators or technology accelerators (3.8), other resources or programs that received support from ACOA (3.8), other ACOA-supported projects undertaken by the client or others, and ACOA-supported trade shows or missions (3.5). [xxxvi]
- Challenges associated with specific projects (e.g. limited budget, implementation delays and difficulties with partnering). Some clients identified insufficient resources as a major constraint to project success, noting that it is very difficult to predict budget needs for R&D projects. Case study participants and key informants noted that, prior to the recent changes, the approval (i.e. all phases of application processing) and contracting processes for AIF was too slow (in one case it took a project two years from application to project start) and the window to apply was too narrow and infrequent, which can make companies more reluctant to apply and can limit the commercial potential of projects. Numerous clients and case study participants identified implementation issues, including internal problems such as issues with management, staff turnover or exceeding budgets) and technical difficulties (i.e. related to research), as impeding the progress of projects. In addition, a few clients and key informants identified coordination issues with respect to partnering with other organizations, such as delays, differences in timelines across partners, and limited interaction between ACOA and some partners or between ACOA regional offices.
Key lessons learned include the importance of being responsive to business needs in program delivery, supporting capacity development, offering a range of assistance to clients that can be tailored to their specific needs, and leveraging resources.
Key informants and case studies identified a number of lessons learned.
- Being responsive to business needs. Half of ACOA key informants noted that it is important to be responsive to business needs in delivering programming (e.g. continuous intake, more timely approval and contracting processes, allow for smaller funding amounts and shorter time frames under the AIF, and flexible program delivery). Case study participants also found that it is important to be responsive to business needs, including: 1) being flexible and responsive to changes in partnering with private-sector organizations given that it is difficult to predict or control changes in market circumstances; 2) moving swiftly and effectively when entering a new market space (several case studies involved the development of novel products/technologies that were expected to occupy a niche market space; however, that space was filled before the product made it to market); and 3) being responsive to the needs of new and emerging sectors. New and emerging clusters may have different needs from more traditional industries and their timelines for commercial success may vary.
- Supporting commercialization capacity development. Some key informants and case study proponents found that ACOA has a role to play in enhancing the commercialization capacity and business skills of clients (i.e. in providing support for innovation ecosystems such as accelerators, incubators, mentorship programs, industry associations and venture capital forums as well as by assisting companies to expand internationally by facilitating participation in trade missions and connecting companies to trade officers).
- Utilizing program tools to leverage private investment. A few key informants noted that tools such as repayable BDP contributions can be useful in leveraging venture capital investment. For example, the ACOA’s regional office in Nova Scotia encourages potential clients to seek venture capital investment based on ACOA’s provision of repayable BDP contributions. These types of arrangements have the potential to result in quality projects (since start-ups receive mentorship support from investors) and increase the likelihood of repayment.
- Recognizing the time, resources and process of innovation. A few key informants noted that the path from innovation to commercialization is complex and can require significant resources and time. Sometimes ACOA assistance will support the first phase or platform in the development of a particular technology. Often clients require further assistance or follow-on investment to adapt technologies to new markets or private-sector partners.
- Aligning with policy trends (e.g. federal, provincial and regional). Key informants also noted that there is a need to align with policy trends in innovation and commercialization. Federal innovation policy (e.g. the Jenkins Report) is placing greater emphasis on business, which aligns well with ACOA’s revised focus on commercialization of innovation.
4.4 Adequacy of performance measurement
ACOA staff have access to project monitoring information as well as some performance measurement data, which is compiled for the purpose of the Departmental Performance Report and/or reported in executive and operational dashboards.
Of the ACOA representatives who were interviewed, most believed that they had access to timely, reliable and relevant project monitoring and performance measurement data. Project level data is available for both the AIF (e.g. quarterly updates, annual reports and monitoring) and BDP (e.g. yearly assessments and monitoring), and ACOA has the ability to roll up high-level information. Using tools such as QAccess and dashboards that summarize QAccess data, information can be accessed on funded projects (e.g. commercial versus non-commercial, sectors, phases of R&D, repayments, partner dollars leveraged, etc.). In addition, representatives indicated that more detailed information can be accessed on a case-by-case basis such as the recent internal review of IC programming. Data is used primarily for monitoring and information gathering (e.g. service delivery levels, delinquencies and trends), managing the portfolio, adjusting the programming and Agency reporting.
Major challenges associated with the existing information system include difficulties in rolling up outcome level data, the time required before impacts are fully realized, difficulties in measuring certain outcomes, and some coding issues. ACOA’s Results Measurement Tracking System, which will allow account managers to track outcomes, holds promise for the availability of outcome data but it is not yet consistently used.
Gaps and challenges identified by key informants included:
- An inability to roll up or standardize progress across innovation and commercialization projects as each project tracks different deliverables, outcomes and paths/timelines.
- Time for impacts to be generated. Projects can take years to fully realize their impacts and project files may be closed prior to those impacts being realized. In addition, the policy focus and metrics may change before the impacts have been fully realized. (Programming used to focus more on R&D capacity building, and those impacts are just starting to be seen; however, the focus has already shifted to commercialization, which has different metrics.)
- Difficulty measuring innovation and commercialization impacts. Innovation and commercialization projects often have spinoff impacts and other indirect or intangible impacts. It is difficult to capture and quantify the full impact of projects (e.g. a project may not result in commercialized technology that was planned but may help the company develop partnerships that result in the development of a different but related technology in the future).
- The need to prioritize the entry of outcome data. Some representatives noted that the time and effort required to enter outcome information into the systems is often prioritized lower than other project monitoring activities. This results in some information not being entered or updated.
Other constraints noted include a lack of sufficient resources to roll up data and track progress (i.e. regions are busy delivering the programming) and certain key indicators (e.g. private capital leveraged) are not currently included in the Dashboard.
In addition, the case studies found some challenges and gaps with respect to performance measurement and information tracking. For most case studies, the project files contained an appropriate level of detail on project descriptions and targeted outcomes. However, several project files were missing information on progress toward outcomes and final reports. ACOA representatives interviewed noted that, in many cases, the final reports existed but were either not loaded into QAccess or were kept in paper format at regional offices. It was noted that the project filing system has been transitioning toward more consistent electronic filing in QAccess.
5.0 Findings: performance – efficiency and economy
This section examines the relationship between program costs and program outputs (operational efficiency) and outcomes (allocative efficiency) as well as how program costs are being minimized while optimizing the achievement of program outcomes (economy).
5.1 Efficient utilization of resources
Salaries and operating costs have averaged about 8.1% of the total IC sub-program budget, which is slightly higher than the previous evaluation period.
As indicated in Table 9, operations and maintenance (O&M) costs accounted for 8.1% of the program budget. O&M costs, as a percentage of total costs, has increased slightly since the last evaluation (from 7.8% for 2005-2006 to 2007-2008 to 8.3% from 2008-09 to 2012-13) due in part to a significant decline in grant and contribution (G&C) expenditures in 2011-12 and, to a lesser extent, an increase in O&M expenditures in 2009-10 and 2010-11 which has since been reversed.
Table 8: Program Delivery Costs by Fiscal Year
Year |
2005-2006 |
2006-2007 |
2007-2008 |
2008-2009 |
2009-2010 |
2010-2011 |
2011-2012 |
2012-2013 |
Total |
||
---|---|---|---|---|---|---|---|---|---|---|---|
Program Expenditures |
|||||||||||
Operating Exp. ($M) |
$1.6 |
$1.4 |
$1.7 |
$1.6 |
$1.6 |
$1.6 |
$1.0 |
$0.7 |
$11.3 |
||
Salary Exp. ($M) |
$4.8 |
$5.4 |
$5.4 |
$5.4 |
$6.7 |
$6.0 |
$5.3 |
$6.0 |
$45.0 |
||
Total O&M Exp. ($M) |
$6.5 |
$6.8 |
$7.1 |
$7.0 |
$8.3 |
$7.6 |
$6.3 |
$6.8 |
$56.2 |
||
G&C Exp. ($M) |
$74.3 |
$82.9 |
$84.2 |
$88.3 |
$89.0 |
$96.2 |
$44.4 |
$81.5 |
$640.9 |
||
Total Expenditures ($M) |
$80.7 |
$89.6 |
$91.4 |
$95.3 |
$97.3 |
$103.8 |
$50.7 |
$88.3 |
$697.1 |
||
Delivery Costs |
|||||||||||
O&M % of Total Exp. |
8.0% |
7.5% |
7.8% |
7.3% |
8.5% |
7.3% |
12.4% |
7.7% |
8.1% |
||
Total # Projects Approved |
150 |
139 |
155 |
149 |
165 |
136 |
125 |
112 |
1,131 |
||
$ O&M per Project Approved |
$43,110 |
$48,654 |
$45,980 |
$46,811 |
$50,203 |
$55,712 |
$50,270 |
$60,470 |
$49,733 |
Source: ACOA GX Financial information management system, data extracted May 2014.
5.2 Structures and mechanisms that support efficiency and economy
Various factors contribute to program efficiency including the governance structure, delivery approach, how resources are used, and use of performance information. Though some factors have constrained efficiency, recent actions have been taken to assess and minimize these barriers.
Most ACOA representatives interviewed perceive that the IC program is (or will be with the new changes resulting from the Innovation Review) designed and delivered efficiently and economically. It is important to note that this evaluation did not re-examine the design and delivery issues studied in the Innovation Review, nor can this evaluation speak to the efficiency and economy gains expected from these changes. Various factors contribute to the efficiency and economy include:
- The regional governance structure. The ACOA governance structure, particularly the focus on regional program delivery, contributes to efficient design and delivery by enabling staff to be closely connected with local stakeholders and clients, and familiar with the local environment and needs. Regional offices tailor strategies to address regional priorities and work closely with provincial governments to align programming, policies, and strategies to develop a greater critical mass of support in targeted areas.
- The delivery approach. Account managers assist clients in navigating the different ways in which ACOA can support their growth and meet their needs at different stages along the innovation continuum (e.g. from product development, to commercialization support, to assistance accessing international markets) and refer them to other services (e.g. incubators, mentorship programs, or other sources of provincial or federal financing).
- Resource leveraging and repayability. The resources dedicated to innovation and commercialization programming are increased through leveraging (each ACOA dollar has been leveraged with $1.32 from other sources) and repayability. Many innovation projects are repayable or conditionally repayable (dependent upon the commercialization of products or achievement of outcomes). Of the IC project funding approved between 2005-2006 and 2012-2013, 80% had been disbursed as of September 2013 of which 31% was a repayable contribution and 8% was conditionally repayable. Of the amount disbursed, 32% of repayable contributions and 6% of conditionally repayable contributions had been repaid; in addition, the percentage will increase over time (e.g. 69% of repayable contributions and 20% of conditionally repayable contributions from 2005-07 have now been repaid).
- The use of performance measures (e.g. dashboards). To help in program management, the dashboards contain useful information on key indicators such as delinquency rates, defaults, portfolio risk ratings, service standards, resource expenditures, and operational data. Plans are being implemented to expand dashboards to include additional data on operations, managing new business, and managing the G&C portfolio.
Key informants, clients, and case study representatives identified that several factors associated with the AIF program constrained program efficiency. These same challenges (e.g. timeliness of AIF approvals and frequency of funding rounds) were also identified during the Innovation Review, which led to the announcement of changes to the programming on July 16, 2014. [xxxvii] Some of the major changes include a move to continuous intake of AIF proposals with bi-monthly or quarterly decision points, increased emphasis on shorter term (i.e., less than three years) AIF projects, increased focus on technically sound projects with commercial potential, integrated branding of innovation programming and intake of project applications, increased promotion of technology adaptation projects, focus on value-added role of industry advisors, increased reliance on regional oversight, and the development of a commercialization element under the BDP.
The results reported in this 2014 evaluation of the IC sub-program are comparable and, in some respects, superior to those reported in the 2010 evaluation.
The 2010 and 2014 used similar methodologies (surveys of clients, case studies, and review of QAccess data), although the 2010 evaluation relied primarily on case studies for assessing economic impacts (e.g. revenues) rather than a mixture of case studies and surveys. In addition, in 2010, certain data (e.g. technology disclosures and spinoffs) were reported only for AIF projects. As indicated in the following table, the two evaluations reported broadly similar data with respect to leverage, revenues and commercialization with the AIF projects in the most recent evaluation reporting higher average figures with respect to patents, transfer agreements, technology disclosures, and spinoff companies.
Table 9: Selected Results from the 2010 and 2014 Evaluations
Impact |
2010 Evaluation |
2014 Evaluation |
---|---|---|
Impacts Per Dollar of ACOA Funding |
||
Leverage 1 |
$1.44 |
$1.32 |
Revenues 2 |
$0.46 |
$0.53 |
Impacts Per Organization Surveyed |
||
Commercialization of technologies, products, processes and services 2 |
2.5 |
2.9 |
Issuance of patents 4 |
0.3 |
0.8 |
Licensing or transfer agreements 4 |
0.4 |
1.2 |
Issuance of technology disclosures 3 |
1.7 |
6.0 |
Development of spinoff companies 3 |
0.2 |
0.8 |
Sources from 2010 and 2014 Evaluation: 1 QAccess Data; 2 2010: Case Studies of 36 projects with revenues allocated over five years; 2014: Client survey; 3 2010: Survey of 88 organizations funded under the AIF; 2014: Client survey of AIF projects only; 4 2010: Survey of 339 organizations funded under the BDP and AIF; 2014: Client survey
5.3 Delivery costs
The IC sub-program compares most closely with the National Research Council’s Industrial Research Assistance Program (IRAP) program, although there are some significant differences. The IC sub-program has comparatively lower delivery costs as a percent of G&Cs provided, although the costs per project approved are similar.
Amongst innovation and commercialization programs, the IC sub-program is most similar to NRC-IRAP, which also employs a regional delivery model in providing G&Cs to businesses to undertake R&D and innovation activities as well as funding for regional and national organizations that support development and commercialization of technologies by SMEs. NRC-IRAP differs from the IC sub-program in that it provides technical and business advisory services and its G&Cs are, on average, smaller and positioned closer to the commercialization end of the innovation spectrum (and, therefore, generate greater impacts in the short-term). These differences are reflected in the costs of the two programs. As indicated below, delivery costs as a percentage of total expenditures are much lower under the IC sub-program (8.3%) than NRC-IRAP (33.4%), although costs per project are similar.
Table 10: Comparison of Delivery Costs, ACOA’s IC Program and NRC-IRAP Program
Program |
ACOA IC |
NRC-IRAP |
---|---|---|
Program Expenditures (2007-2008 to 2011-2012) |
||
Total O&M Exp. ($M) |
$36.2 |
$222.7 |
Total G&C Exp. ($M) |
$402.2 |
$444.5 |
Total Expenditures |
$440.4 |
$667.2 |
Delivery Costs |
||
O&M as a Per cent of Total Exp. |
8.3% |
33.4% |
Total Projects Approved |
730 |
5,197 |
O&M per Project Approved |
$49,652 |
$42,843 |
5.4 Alternative modes of delivery
The most common suggestions to maintain or improve program efficiency and economy were to streamline processes, reduce timelines for approval, and pursue a more integrated approach to program delivery, which are in line with the changes currently being made to the programming.
Other common suggestions were to:
- Ensure that the programming remains flexible and responsive to client needs. Clients, case study representatives and key informants stressed the importance of flexibility and responsiveness, with some suggesting changes such as allowing for more than one project to be supported at the same time, broadening the definition of eligible costs, and reducing reporting requirements for smaller projects.
- Continue to support commercialization capacity building. Key informants, clients and case study representatives highlighted the importance of support mechanisms such as providing support to non-profit organizations and events that are part of the innovation eco-system, providing direct assistance to SMEs to allow them to access skilled expertise, and facilitating partnerships between early stage SMEs and key players in global networks.
- Further strengthen communication and alignment with partners. Key informants, clients, and case study representatives noted that ACOA needs to maintain close coordination and communication with partners in order to reduce the potential for duplication, assist clients with path finding and leveraging of funding, coordinate support to strategic sectors, share information about the available programs, and play a convening role to promote cross-provincial coordination in pursuing pan-Atlantic regional objectives.
- Develop better mechanisms to roll up and report on project outcomes. Several groups of ACOA representatives suggested that ACOA should develop better mechanisms to roll-up project outcomes, employ a broader range of data sources (e.g. project reports, surveys, and anecdotal evidence via case studies) on an ongoing or annual basis, revise the performance indicators (e.g. track both standard and unique indicators, report on key success factors, and monitor trends in macro-economic indicators, etc.), and improve its systems and tools (e.g. QAccess and dashboards).
- Further support ACOA staff in program delivery (e.g. facilitate ongoing sharing of best practices in delivery approaches, training, and capacity building). Key informants noted that there are opportunities to further share and benefit from best practices developed in different ACOA regions. It was noted that processes that brought staff from different regions together (e.g. for training or to participate in AIF sector teams) helped staff to gain knowledge and make connections. Given the increasing complexity of projects, key informants and clients noted that ACOA staff should receive more training related to innovation and commercialization (e.g. training on IP agreements), and a few key informants noted that account managers should be provided with more guidance and training with respect to due diligence (at the contract and implementation stages), understanding transactions and reading financial statements.
6.0 Conclusions and recommendations
6.1 Conclusions
The major conclusions arising from the evaluation of the Innovation and Commercialization sub- program are as follows.
Relevance
- There is a strong, continued need for the IC sub-program given the importance of innovation and commercialization to regional competitiveness, the need to build on the progress made in recent years, and the fit between the program and client needs.The IC sub-program responds to this need by committing significant funding for innovation and commercialization and by placing emphasis on the development of SMEs and clusters in key strategic sectors. Most clients indicated that the assistance provided by ACOA met the needs of their organization and that only 4% of projects would have proceeded in the absence of ACOA support.
- Innovation and commercialization are major drivers of business productivity and economic competiveness. Although Atlantic Canada, and more specifically the IC sub-program, has made progress in strengthening commercialization capacity and increasing investment in R&D over the past decade, more needs to be done. The region continues to lag behind other regions of Canada on key innovation and commercialization indicators (e.g. level of investment in R&D, level of R&D performed by the private sector, access to venture capital and highly qualified personnel, proportion of federal research grant funding awarded, etc.). Factors that constrain innovation performance include the industrial structure, the small domestic market, significant interprovincial and regional disparities, and a risk averse business culture.
- Changes have recently been made to better align the IC sub-program with client needs and evolving federal government priorities. These changes may impact demand for funding, types of clients and projects, results and efficiency.
- The Agency has access to the information it needs to adjust its programming and strategies in response to changes in the innovation and commercialization landscape. ACOA has monitored the changing innovation and commercialization landscape in a recent review of innovation programming and adjusted its programming to better meet existing and emerging needs, as well as reflect evolving federal government priorities (e.g. changes to programming with a renewed focus on meeting the needs of businesses, such as more timely delivery, and a greater focus on funding commercialization).
- The IC sub-program tends to complement rather than duplicate innovation and commercialization programming delivered by other organizations as well as other ACOA sub- programs due to its focus on commercialization, levels of assistance, ability to take on risk, regional delivery and comprehensive support. Given regular changes in the programming environment, ensuring that the IC sub-program is well coordinated requires ongoing effort.The IC sub-program also works closely with other ACOA programming in providing direct and indirect support to clients along a continuum of development. Just over half of IC clients received funding from more than one ACOA sub-program.
- The IC sub-program is differentiated from other programs that promote innovation and commercialization in Atlantic Canada by its strong focus on commercialization, levels of financial assistance available, support for riskier projects and clients, and regional delivery. To minimize the potential for overlap and to leverage support for clients, ACOA coordinates activities with other programs. Seventy-five per cent of clients reported their projects received funding from sources other than ACOA. Although some areas of overlap were identified by key informants (e.g. some provincial governments are developing programming similar to ACOA’s), they were seen as minimal and normally addressed through communication and collaboration.
- The IC sub-program is consistent with the strategic priorities, roles and responsibilities of the federal government and ACOA as well as the strategies and approaches implemented by federal organizations in other regions of Canada and in other jurisdictions.The IC sub-program aligns well with the roles and responsibilities of ACOA and the federal government in Canada as well as other jurisdictions. The sub-program is consistent with the ACOA Act, which established the Agency as the primary federal government department responsible for economic development efforts in Atlantic Canada. The focus, priorities and roles of the sub-program are also consistent with the approaches implemented by federal government organizations in other jurisdictions (e.g. placing a high priority on innovation and commercialization, shifting their emphasis toward market “pull” innovation policy, broadening their definition of innovation, developing clusters, placing greater priority on SMEs, promoting partnerships between the private sector and research organizations, addressing commercialization gaps, and coordinating programming across different levels of government and partners).
- The objectives of the IC sub-program are consistent with federal government priorities, particularly as stated in the Science and Technology Strategy 2007, Budget Plan 2014, speeches from the Throne and the 2011 Jenkins Report. ACOA has taken steps to ensure its programming aligns with federal government priorities by promoting public-private partnerships, targeting priority sectors and increasing the focus on commercialization. The objectives of the sub-program are also consistent with ACOA’s focus on helping Atlantic SMEs to be more innovative, productive and competitive.
Performance – effectiveness
- The IC sub-program has resulted in extensive direct and leveraged investments in innovation and commercialization projects which, in turn, have generated a wide range of impacts in the immediate and intermediate term. Over the long term, the sub-program has expanded innovation and commercialization capacity and contributed to the growth and competitiveness of SMEs in Atlantic Canada.
- An average of over $80 million in project funding was approved annually by ACOA, and that amount was leveraged with $1.32 in funding from other sources for every dollar contributed by ACOA. On an annual basis, the IC sub-program has stimulated approximately $131 million in applied R&D expenditures, including investments of $38 million in research-related equipment and infrastructure, $57 million in commercialization expenditures, project employment for over 1,400 people, the development of 547 partnerships and 405 new technologies, products or processes, the adaptation of 370 technologies, and the commercialization of 521 technologies, products or processes that, as per the latest fiscal year, had generated $48 million in direct revenues and $26 million in spinoff revenues as well as 750 direct jobs and 208 spinoff jobs. These reported results are understated in that the impacts will continue to grow as more projects reach the commercialization stage and revenues and employment from commercialization increases over time. On an annual basis, the IC sub-program has also led to 114 patents, 212 technology disclosures, $66 million in further investment in R&D, $26 million in cost savings or productivity improvements, and the attraction of $110 million in new investment. Over the long term, ACOA funding of projects has supported the establishment and further development of various industrial clusters in Atlantic Canada, including aerospace and defence, life sciences and biotechnology, ocean technologies, information and communications technology, and energy (particularly renewable energy) sectors. The economic analysis, using the CBoC’s econometric model, estimates that the IC sub-program generated an increase in GDP of $5.49 for every dollar expended by ACOA over the 2007-2008 to 2012-2013 period.
- Major factors that influence the progress made toward expected outcomes include: the characteristics of projects; the incidence and magnitude of highly successful projects; economic conditions and the competitive environment; the client’s business practices, knowledge and capabilities; the design and delivery of the sub-program; the interrelationship between investments made by ACOA; and project-specific challenges.The main factors that constrained progress include challenges associated with the project (e.g. lack of budget, implementation delays, difficulties with partnering), inefficiencies in program delivery (e.g. one entry date and delayed contracting times for AIF projects), and economic, social and geographic challenges (e.g. global competition and the pace of change, lack of available financing, lack of capacity for applied R&D, lack of skilled personnel, geographic distance to markets, and a risk averse business culture).
- Progress is influenced by project type, focus (e.g. projects that are more focused on commercialization generate a greater return per dollar invested), timing (e.g. impacts increase over time), sector (e.g. biotechnology projects tend to have longer commercialization time frames than information and communications technology projects), the client’s own business practices (e.g. technical and business management skills), the design and delivery of ACOA programming (e.g. comprehensive, flexible, low-risk support that promotes leveraging and industry involvement), and the economic environment (e.g. market demand and the availability of resources).
- Key lessons learned include the importance of being responsive to business needs in program delivery, supporting capacity development, offering a range of assistance to clients that can be tailored to their specific needs, and leveraging resources.
- Major lessons learned during the implementation of projects and programming included the importance of being responsive to business needs (e.g. being flexible and responsive to change in partnering with private-sector organizations, moving swiftly and effectively when entering a new market, and being responsive to the needs of new and emerging sectors), supporting commercialization capacity building (e.g. supporting innovation and commercialization ecosystems such as accelerators, mentorship programs, industry association and venture capital forums), utilizing program tools to leverage private investment (e.g. the approach of the Nova Scotia regional office in using repayable BDP to leverage venture capital investment for projects), recognizing that innovation can be a lengthy and complex process, and aligning with partners and policy trends (e.g. federal, provincial and regional).
- While ACOA staff have access to project monitoring and performance reporting information, there are some specific challenges that need to be addressed (e.g. inability to roll up outcome data and difficulties in coding impacts).The main gaps and challenges associated with performance measurement include a lack of ability to roll up or standardize progress or outcomes across the portfolio; time to see impacts; difficulty in measuring innovation and commercialization impacts (e.g. difficulties in coding projects to innovation versus other programs such as productivity and growth); and the relative priority afforded to the entry of outcome data in the QAccess system
- Most ACOA representatives interviewed reported having access to timely, reliable and relevant project monitoring and performance measurement data in terms of project-level data (e.g. quarterly updates, annual reports and risk monitoring) and high-level information using the dashboards and QAccess (e.g. sectors, repayments, partner dollars leveraged, etc.). More detailed information can be accessed on a case- by-case basis and through more thorough reviews. Data is used primarily for monitoring and information gathering (e.g. service delivery levels, delinquencies, trends), managing the portfolio (e.g. ensuring balance in terms of level of risk), adjusting the programming, and Agency reporting.
Performance – economy and efficiency
- The existing governance structure, delivery model and funding mechanisms appear to support the efficient and economical delivery of the programming (O&M costs averaged about 8% of total program expenditures). Further improvements are expected from recent changes to the programming. There may be opportunities to further improve efficiency and economy by sharing tools, processes, promising practices and lessons learned across regions.Salaries and operating costs, as a percentage of the total program budget for the sub-program total costs, have increased slightly since the last evaluation (from 7.8% for 2005-2006 to 2007- 2008, to 8.3% from 2008-2009 to 2012-2013). This is due, in large part, to a significant decline in G&C expenditures in 2011-12. There may be opportunities to further improve delivery across regions (e.g. through standardization and the sharing of tools, processes, promising practices and lessons learned).
- Various factors contribute to the IC sub-program’s efficiency, including the regional governance structure (e.g. allows regions to tailor strategies and focus to needs), the client-centric delivery approach (e.g. offering a range of support mechanisms), the increased availability of resources through leveraging, the sharing of risks through repayability, and the collection and use of high-level performance information (e.g. dashboards) to monitor portfolio performance and make investment decisions.
- Actions have recently been taken to address program barriers. While the changes are expected to benefit efficiency and economy, the results will need to be carefully monitored.
- The main efficiency constraints identified relate to the timeliness of approvals (i.e. at all phases of application processing) and contracting process for the AIF (e.g. it can take too long for AIF projects to be approved and contracted, the window to apply for funding is too small, and the funding rounds are too infrequent). An internal review was recently undertaken of innovation programming and, as of July 2014, changes have been implemented that aim to improve program efficiency (e.g. moving to a continuous intake system for the AIF, using a generic Agency application form for both the AIF and BDP, lowering the AIF private-sector threshold, and introducing a new BDP commercialization element). Plans for measuring the impact of these changes are being implemented.
6.2 Recommendations
The major recommendations arising from the evaluation of the Innovation and Commercialization sub-program are as follows.
- Carefully monitor the recent programming changes to assess the impact and determine the need for further changes or adjustments.Some changes have also been made to the delivery structure. One of the benefits of the regional structure and level of regional autonomy that exists within ACOA is the opportunity it provides to share experiences, knowledge and information gained with respect to effective approaches, client needs, recent developments, lessons learned and promising practices. This information, however, is often not formalized, with preference given to informal means of communication. The regional structure can create silos between offices. The AIF sector committees provided a valuable venue for the sharing of information between account managers and between the regions overall. However, with the restructuring of the AIF program, these sector committees are no longer in place. A key function of the new and existing governance structures should be to facilitate and strengthen internal communication and formalize the sharing of information and lessons learned across regions. Opportunities should also be explored for facilitating formal information sharing across the sub-programs within ED.
- The impacts of the recent programming and structural changes should be carefully monitored to assess the impact on the demand for funding, the characteristics of clients and projects supported (e.g. the relative balance between investments in R&D and investments in commercialization capacity), approval times, risk, resulting impacts, and ongoing communication and sharing of information, both internally and externally, regarding client needs, recent developments, lessons learned and promising practices. Existing and new governance structures at the project level (e.g. commercialization champions, use of industry advisors and technical advisors in conducting due diligence, etc.) and at the strategic level (e.g. DG Operations Committee, ED Directors Committee, AIF Advisory Board, etc.) will assist in monitoring and assessing the risks and impacts of the changes.
- ACOA has recently introduced significant changes to the IC sub-program. Based on the results of its recent review, the Agency renewed its commitment to IC and is strengthening its programming by launching a new BDP commercialization funding element as well as by moving to a continuous intake system and lowering the minimum funding threshold to better enable the AIF to meet the needs of business. With these changes, the programming is expected to better operate at the speed of business and to place a greater emphasis on commercialization (without excluding targeted investments in R&D capacity).
- Revise the Performance Measurement Strategy (PMS) for the IC sub-program to reflect the recent program changes, and improve the utility of the system for both performance reporting and management planning by integrating outcome data into monitoring systems.To address these issues, ACOA should consider employing a mixture of standard and unique indicators. (A few core, high-level performance indicators can be applied to each IC project; these standard indicators can be augmented, as necessary, by additional project-specific indicators.) Consideration should also be given to moving toward a more client-focused approach to performance measurement with respect to project outcomes (rather than a strictly project-by-project approach), aligned with ACOA’s client-centred approach to ED program delivery. This approach would reflect the full range of assistance provided to a client, recognizing that the majority of IC clients receive ACOA support multiple times and under multiple sub-programs. ACOA should also explore opportunities to mine the existing data systems to monitor performance, assess trends and inform future programming decisions. Existing data could be augmented by collecting and rolling up information on client needs or the progress of clients in achieving milestones (e.g. from development to commercialization) as part of client tracking activities.
- Program logic models are being revised and the results will need to be reflected in a revised PMS. Difficulties in meaningfully rolling up project outcomes have meant that the impact information reported by clients has not been used extensively for either performance reporting or management planning. A low priority is often placed on inputting impact or outcome information reported by clients into QAccess.
Appendix A: management action plan
Evaluation |
Recommendation |
Planned Actions |
Responsibility |
Target Date |
Status |
---|---|---|---|---|---|
Innovation and Commercialization Management Action Plan (MAP) approval date: Pending |
|
The Agency will monitor programming changes and their impacts and will ensure ongoing communication by making use of the existing reporting mechanisms and governance structure, including the Pan Agency Review Committee (PARC), the director deneral of Operations (DG Ops), directors of Enterprise Development (ED), Atlantic Innovation Fund (AIF) regional coordinators and the Commercialization Champions Committee. More specifically:
The Agency will complete a risk assessment exercise on the new AIF due diligence process. |
Director general, ED in collaboration with the director general, Policy, the regional and head office director generals and ED directors. Director, Innovation and Entrepreneurship |
Ongoing monitoring of the planned actions to ensure their effectiveness will occur over a two-year period ending December 31, 2016. March 31, 2015 |
Plans are in place for monitoring impacts resulting from the implementation of the recent programming changes and to ensure ongoing communications with internal and external stakeholders. The assessment exercise is in the planning stages |
|
In Spring 2014, ACOA engaged Performance Management Network to assist in undertaking a review of the Performance Measurement Strategy for ED and its sub programs. Work will continue internally to refine and finalize the PMS, taking into consideration the findings of the evaluation as well as Agency plans regarding performance measurement for ED moving forward. | Director, Innovation and Entrepreneurship in collaboration with regional and head office ED directors. | October 31, 2015 | The PMS review is currently under way. |
Appendix B: overview of ACOA
Program |
2012-2013 Actual Spending ($M) |
Sub-program |
Key Indicators of Performance |
---|---|---|---|
Enterprise Development |
$179.9 |
|
|
Community Development |
$88.5 |
|
|
Policy, Advocacy and Coordination |
$12.4 |
|
|
Internal Services |
$34.9 |
|
|
Total |
$315.7 |
|
Sources: Atlantic Canada Opportunities Agency 2012-13 Departmental Performance Report and 2013-14 Report on Plans and Priorities.
Appendix C: evaluation issues and questions
Evaluation Issues and Questions |
|
---|---|
Relevance |
|
Issue 1: Continued Need for the Program |
|
1.1 |
To what extent does the Innovation and Commercialization sub-program continue to address a demonstrable need? |
1.2 |
To what extent is the sub-program aligned with the existing and emerging needs of its stakeholders? |
1.3 |
What other mechanisms exist to address these needs? Where are the areas of integration, overlap, duplication and gaps? |
Issue 2: Alignment with Government Priorities |
|
2.1 |
To what extent is the sub-program aligned with federal government priorities and expectations? To what extent is it aligned with ACOA’s strategic outcome, priorities and strategy? |
Issue 3: Alignment with Federal Roles and Responsibilities |
|
3.1 |
To what extent is the sub-program aligned with federal roles and responsibilities? |
Performance |
|
Issue 4: Effectiveness |
|
4.1 |
How and to what extent is the sub-program achieving expected outcomes (immediate, intermediate and long-term outcomes)? What has been the contribution of this sub-program to the innovation ecosystem of Atlantic Canada? |
4.2 |
To what extent has ACOA’s gradual transition toward a greater focus on commercialization (e.g. in project selection) contributed to the achievement of expected results? |
4.3 |
Incrementality: What impact would the absence of the sub-program have on projects/initiatives? |
4.4 |
What are the facilitators and barriers to achieving expected outcomes (from ACOA and client perspective)? To what extent are these being addressed? |
4.5 |
What lessons have been learned in the implementation of the components of the sub- program? How can these lessons contribute to future and/or other programming? |
4.6 |
What unintended outcomes have been achieved? |
4.7 |
To what extent are the sub-program’s performance measurement and reporting structures effective in reporting on the achievement of outcomes? How and to what extent is the performance information used by clients and by ACOA? |
Issue 5: Efficiency and Economy |
|
5.1 |
In the context of the results being achieved, how and to what extent are the resources allocated to the sub-program efficiently and economically utilized? |
5.2 |
Are there more efficient and economical ways of achieving expected results, taking into consideration current and alternative approaches to delivery, sub-program governance, barriers to efficiency/economy, and the application of best practices and lessons learned? |
Appendix D: innovation and commercialization sub-program logic model
Reach |
CLIENTS / DIRECT TARGET Universities Firms Research institutions OTHER STAKEHOLDERS Industry groups / associations Federal science-based departments National programs (e.g. CFI, NSERC, TPC) Provincial governments CO-DELIVERERS Advisory Boards Science / Industry advisors |
Activities and Outputs |
Financing innovation / commercialization projects |
---|---|---|---|
Immediate / Direct Outcomes |
Increased applied R&D activity Increased capacity for applied R&D Increased adaptation of technology Establishment of networks and of meaningful partnerships/alliances Attraction, development and retention of R&D staff Increased capacity [e.g. knowledge, skills, infrastructure] for commercialization |
||
Intermediate / Indirect Outcomes |
Development of new or improved technologies / products / processes / services [Benefits of] increased applied research capacity in Atlantic Canada in both the private sector and institutions Research results are commercialized with benefits to Atlantic Canada |
||
Long-term Outcome |
Strengthen Atlantic Canada’s innovation and commercialization capacity |
||
Enterprise Development |
Improved growth and competitiveness of Atlantic SMEs |
||
Strategic Outcome |
A competitive Atlantic Canadian economy |
Adapted from: ACOA, Performance Measurement. ACOA 2012-2013 Program Activities and Sub-activities Logic Models, February 2012.
Appendix E: other programs supporting innovation and commercialization in Atlantic Canada
Organization |
Description |
Types of Support |
Target Groups |
Resources and Funding Amounts |
---|---|---|---|---|
FEDERAL GOVERNMENT PROGRAMS |
||||
National Research Council of Canada Industrial Research Assistance Program (NRC-IRAP) |
|
|
|
|
Natural Sciences and Engineering Research Council (NSERC) |
|
|
|
|
Canadian Institutes of Health and Research (CIHR) |
|
|
|
|
Social Sciences and Humanities Research Council (SSHRC) |
|
|
|
|
Networks of Centres of Excellence (NCE) |
|
|
|
|
Canada Foundation for Innovation (CFI) |
|
|
|
|
Business Development Bank of Canada (BDC) |
|
|
|
|
Canada Revenue Agency (CRA) |
|
|
|
|
Industry Canada |
|
|
|
|
Department of Finance |
|
|
|
|
Public Works and Government Services Canada |
|
|
|
|
Sustainable Development Technology Canada (SDTC) |
|
|
|
|
PROVINCIAL GOVERNMENT PROGRAMS |
||||
Innovacorp |
|
|
|
|
New Brunswick Innovation Foundation (NBIF) |
|
|
|
|
Nova Scotia Department of Economic and Rural Development and Tourism |
|
|
|
|
Innovation PEI |
|
|
|
|
Research and Development Corporation (N.L.) |
|
|
|
|
New Brunswick Department of Economic Development |
|
|
|
|
Regional Development Corporation (N.B.) |
|
|
|
|
RPC |
|
|
|
|
OTHER NOT-FOR-PROFIT / REGIONAL PROGRAMS |
||||
Springboard (Atlantic Canada) |
|
|
|
|
BioNova (N.S.) |
|
|
|
|
BioNB (N.B.) |
|
|
|
|
The Next Phase (N.S.) |
|
|
|
|
Propel ICT/ Launch 36 (N.B.) |
|
|
|
|
Volta – Halifax Startup House (N.S.) |
|
|
|
|
Venn Innovation Inc. (formerly Tech South East) (N.B.) |
|
|
|
|
Wallace McCain Institute (N.B.) |
|
|
|
|
Pond-Deshpande Centre for Innovation & Entrepreneurship (N.B.) |
|
|
|
|
Genesis Centre (N.L.) |
|
|
|
|
Appendix F: client survey ratings tables
Table 11: Importance of Factors When Considering Applying to ACOA versus Another Source
Question: On a scale of 1 to 5, where 1 is not at all important and 5 is very important, how important were the following considerations in your decision to apply for funding from ACOA rather than from another organization?
Question |
BDP Non-commercial |
BDP Commercial |
AIF Non-commercial |
AIF Commercial |
Average |
---|---|---|---|---|---|
Level of funding available |
4.5 |
4.6 |
4.9 |
4.9 |
4.7 |
Eligibility requirements |
4.3 |
4.3 |
4.2 |
4.3 |
4.3 |
Timeliness of project approvals |
4.1 |
4.3 |
4.1 |
4.0 |
4.3 |
Definition of eligible expenditures |
4.4 |
4.3 |
4.2 |
3.8 |
4.2 |
Unsuccessfully approached other funding sources |
3.7 |
4.1 |
3.9 |
3.9 |
4.1 |
Other funding sources were not available |
4.2 |
4.0 |
4.1 |
3.7 |
4.0 |
Familiarity with ACOA |
4.4 |
4.0 |
3.6 |
3.9 |
4.0 |
The application process |
3.2 |
3.5 |
3.6 |
3.4 |
3.5 |
Reporting requirements |
3.2 |
3.2 |
3.5 |
3.1 |
3.2 |
Non-financial assistance provided by ACOA |
3.3 |
3.3 |
3.1 |
2.9 |
3.2 |
Table 12: Ratings of Immediate Impacts Generated by Undertaking the Project
Question: To what extent did undertaking the ACOA-assisted project result in: (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent)
Question |
BDP Non-commercial |
BDP Commercial |
AIF Non-commercial |
AIF Commercial |
Total |
---|---|---|---|---|---|
Improving business management practices used by your organization |
3.2 |
3.3 |
3.3 |
2.9 |
3.2 |
Improving the marketing efforts of your organization |
3.1 |
3.3 |
3.1 |
2.7 |
3.2 |
Strengthening the research capacity of your organization (e.g. human resources, facilities and equipment) |
3.9 |
3.3 |
4.3 |
4.0 |
3.5 |
The development of new partnerships and alliances between your organization and others |
4.3 |
3.5 |
4.3 |
3.5 |
3.7 |
The expansion of your existing networks, partnerships and alliances |
4.2 |
3.5 |
4.2 |
3.6 |
3.7 |
Strengthening the capacity of your organization to commercialize new technologies, products, processes or services (e.g. knowledge, skills and infrastructure related to commercialization) |
3.7 |
3.8 |
3.5 |
3.7 |
3.7 |
The adaptation of existing technology for use by your organization or others |
4.2 |
3.7 |
4.4 |
3.1 |
3.8 |
Increasing the human resource development capacity of your organization |
4.5 |
3.8 |
4.4 |
4.0 |
3.9 |
The development of the skills and abilities of any students who worked on the project |
4.8 |
4.1 |
4.7 |
4.5 |
4.3 |
The development of the skills and abilities of your staff |
4.6 |
4.3 |
4.6 |
4.4 |
4.3 |
Table 13: Ratings of Intermediate Impacts Generated by Undertaking the Project
Question: To what extent did undertaking the ACOA-assisted project result in: (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent)
Question |
BDP Non-commercial |
BDP Commercial |
AIF Non-commercial |
AIF Commercial |
Total |
---|---|---|---|---|---|
The establishment of one or more spinoff companies |
1.8 |
1.4 |
2.0 |
1.2 |
1.5 |
Filing technology disclosures |
1.9 |
1.3 |
2.8 |
2.3 |
1.6 |
Filing new patents |
1.8 |
1.6 |
2.3 |
1.9 |
1.7 |
Licensing arrangements or transfer agreements signed between your organization and others |
1.8 |
1.8 |
2.6 |
2.1 |
1.9 |
Further or ongoing investment by your organization in R&D |
2.9 |
2.7 |
2.6 |
3.4 |
2.8 |
The commercialization of new technologies, products, processes or services by your organization or others |
3.0 |
3.5 |
3.1 |
3.4 |
3.4 |
The development of new or improved technologies, products, processes or services |
4.1 |
4.1 |
4.5 |
4.5 |
4.2 |
Table 14: Ratings of Long-term Impacts Generated by Undertaking the Project
Question: To what extent did undertaking the ACOA-assisted project result in: (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent)
Question |
BDP Non-commercial |
BDP Commercial |
AIF Non-commercial |
AIF Commercial |
Total |
---|---|---|---|---|---|
Further investment in your organization by others |
2.8 |
1.8 |
2.6 |
2.0 |
2.0 |
Social benefits such as environmental, safety or health benefits |
2.2 |
2.3 |
2.7 |
2.5 |
2.3 |
Cost savings and/or improvements in productivity for your organization |
1.9 |
2.8 |
1.7 |
2.0 |
2.5 |
Improvements in your organization’s competitive position (e.g. market share, revenues or profitability) |
2.8 |
3.5 |
2.5 |
3.1 |
3.3 |
Table 15: Use and Ratings of Other Resources Supported by ACOA
Question: Did your organization make use of any of the following resources supported by ACOA when implementing the project or acting upon the results? If so, to what extent did the resource contribute to the success of your project and the resulting impacts? (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent)
BDP Comm. (n=202) |
BDP Non-comm. (n=28) |
AIF Comm. (n=24) |
AIF Non-comm. (n=23) |
Total (n=277) |
||||||
---|---|---|---|---|---|---|---|---|---|---|
% used service |
Rating |
% used service |
Rating |
% used service |
Rating |
% used service |
Rating |
% used service |
Rating |
|
Incubators or technology accelerators |
9% |
4.0 |
25% |
3.4 |
8% |
3.0 |
13% |
4.3 |
11% |
3.8 |
Other resources or programs that received support from ACOA |
20% |
3.7 |
32% |
4.1 |
17% |
3.0 |
17% |
4.0 |
21% |
3.8 |
Other ACOA-supported projects undertaken by your organization or others |
25% |
3.6 |
54% |
4.1 |
21% |
3.4 |
22% |
4.0 |
27% |
3.7 |
ACOA-supported trade shows or missions |
20% |
3.6 |
32% |
3.3 |
42% |
3.1 |
26% |
3.2 |
23% |
3.5 |
Learnsphere (The Commercialization Consulting and Mentoring Program) |
6% |
3.4 |
4% |
3.0 |
21% |
3.4 |
13% |
3.7 |
8% |
3.4 |
Other services by Springboard Atlantic |
6% |
2.6 |
18% |
4 |
4% |
2.0 |
30% |
3.9 |
9% |
3.2 |
Technology transfer office/ industry liaison offices through Springboard Atlantic |
16% |
1.9 |
21% |
4.2 |
8% |
2.0 |
39% |
4.0 |
18% |
2.6 |
Appendix G: profile of IC projects, by program and type of client, 2005-2006 to 2012-2013
AIF |
BDP |
|||
---|---|---|---|---|
Commercial |
Non-Commercial |
Commercial |
Non-Commercial |
|
Projects |
90 |
89 |
790 |
162 |
8% |
8% |
70% |
14% |
|
Unique Clients |
24 |
52 |
610 |
76 |
Leading Types of Clients |
|
|
|
|
ACOA Funding |
$209.5 million |
$204.9 million |
$198.0 million |
$40.1 million |
32% |
31% |
30% |
6% |
|
Average ACOA Assistance |
$2.33 million |
$2.30 million |
$250,686 |
$247,766 |
Total Costs |
$437.9 million |
$463.0 million |
$499.1 million |
$112.8 million |
Avg. Project Budget |
$4.87 million |
$5.20 million |
$631,766 |
$696,030 |
Leverage Rate |
$1.09 |
$1.26 |
$1.52 |
$1.81 |
Incrementality (per client survey) in the absence of funding: |
Project would have
|
Project would have:
|
Project would have:
|
Project would have:
|
Sources of Funding |
|
|
|
|
Average Length |
4.1 years |
5.0 years |
1.3 years |
1.2 years |
Sub-Elements Funded (% of ACOA funding) |
|
|
|
|
Expenditures by Category (as % of total project costs) |
|
|
|
|
Percent of Project Budget Spent On (per client survey): |
||||
R&D |
79% |
88% |
39% |
35% |
Commercialization |
14% |
10% |
45% |
35% |
Locale of R&D Activities (per client survey): |
||||
In-house |
77% |
71% |
74% |
74% |
External |
5% |
5% |
14% |
22% |
Both |
18% |
24% |
13% |
4% |
Sources: QAccess data and client survey
Footnotes
[i] Atlantic Canada Opportunities Agency, 2012-2013 Departmental Performance Report.
[ii] Government of Canada, Budget 2014. http://www.budget.canada.ca/2014/docs/plan/ch3-2-eng.html.
[iii] Innovation and Commercialization full- time equivalent data was not available for 2005-2006 and 2006-2007.
[iv] The economic and community development activities formerly undertaken by Enterprise Cape Breton Corporation have transitioned over to ACOA. However, given the timing and scope of the present evaluation, Cape Breton is reported as a separate region.
[v] The delivery process described applies to the period being evaluated. Between fall 2012 and spring 2014, ACOA’s Policy and Programs branch conducted a formal, ten-year retrospective analysis of the Agency’s IC activities and coordinated consultations with stakeholders across Atlantic Canada. Based on the results, the Agency renewed its commitment to IC and is strengthening its programming by launching a new BDP commercialization funding element as well as by streamlining the BDP and AIF application intake (one application form), changing the AIF approval process to a continuous intake system and lowering the minimum funding threshold to better enable the AIF to meet the needs of business. These changes are expected to impact the demand for funding, the types of clients and projects supported (i.e. increased emphasis on commercialization), the resulting impacts and efficiency.
[vi] OECD/Eurostat, Oslo Manual: Guidelines for Collecting and Interpreting Innovation Data, 2005.
[vii] World Economic Forum, The Global Competitiveness Report 2013–2014.
[viii] Government of Canada, Innovation Canada: A Call to Action. Review of Federal Support to Research and Development – Expert Panel Report, October 17, 2011.
[ix] RE$EARCH Infosource Inc., 2011
[x] Statistics Canada, Table 358-0161 – Business enterprise research and development (BERD) characteristics, by industry group based on the North American Industry Classification System (NAICS), provinces and territories, annual.
[xi] Some conventional indicators such as patenting cover only a small portion of the innovation spectrum. In recognition of this, discussions by the federal government are shifting to a new paradigm of innovation. This point is reviewed further in a discussion of alignment with federal roles and responsibilities.
[xii] According to a review of online grant award databases, the level of funding awarded to Atlantic institutions has increased from about $30 million in 2001-2002 to $50 million in 2010-2011 for Natural Sciences and Engineering Research Council (NSERC) grants and scholarships; from $8.7 million in 1999-2000 to $24.8 million in 2013-2014 for Canadian Institutes of Health Research (CIHR) grants and awards; from $22.9 milion in 2001-2002 to $43.0 million in 2012-2013 for Social Sciences and Humanities Research Council of Canada (SSHRC) funding; and from $0 to $33.7 million from the Networks of Centres of Excellence (NCE) to support three centres of excellence over four to five years. However, Atlantic Canada’s share of national funding has remained at about 5% of total NSERC expenditures, 3% of total CIHR funding in Canada; and 6.5% of total SSHRC expenditures in Canada.
[xiii] Statistics Canada, CANSIM table 358- 0001. Gross domestic expenditures on research and development, by science type and by funder and performer sector (current prices).
[xiv] NSERC: Natural Sciences and Engineering Research Council; NRC-IRAP: National Research Council of Canada Industrial Research Assistance Program; CIHR: Canadian Institutes of Health Research; NCE: Networks of Centres of Excellence; CFI: Canada Foundation for Innovation; BDC: Business Development Bank of Canada.
[xv] Currently called Department of Business, Tourism, Culture and Rural Development.
[xvi] Mobilizing Science and Technology to Canada’s Advantage, 2007
[xvii] Government of Canada, Budget 2014. http://www.budget.canada.ca/2014/docs/plan/ch3-2-eng.html.
[xviii] Government of Canada, Speech from the Throne to open the Second Session of the Forty-first Parliament of Canada, October 16, 2013. House of Commons Journals: 2013.10.16, vol. 147. https://lop.parl.ca/sites/ParlInfo/default/en_CA/Parliament/procedure/throneSpeech/speech412.
[xix] Government of Canada, Speech from the Throne to open the First Session of the Forty-first Parliament of Canada. June 3, 2011. House of Commons Journals: 2011.06.03, vol. 146.
[xx] Industry Canada, Innovation Canada: A Call to Action. Review of Federal Support to Research and Development – Expert Panel Report. http://publications.gc.ca/site/eng/404471/publication.html
[xxi] Government of Canada, Constitution Act, 1867 to 1982, Part III, Equalization and Regional Disparities, section 36. (1) (b). http://laws- lois.justice.gc.ca/eng/Const/page-16.html#h-53
[xxii] Government of Canada, Atlantic Canada Opportunities Agency Act, [Enacted as Part I to R.S.C. 1985, c. 41 (4th Supp.), in force September 15, 1988. https://laws-lois.justice.gc.ca/eng/Const/page-16.html#h-53
[xxiii] Speech from the Throne, October 2013
[xxiv] The $400-million Venture Capital Action Plan, announced in January 2013, will establish up to two new large-scale private-sector-led national funds with $250 million. The Canada Accelerator & Incubator Program, to be administered by NRC-IRAP, will help establish a critical mass of incubators and accelerators. The Technology Demonstration Program provides non-repayable contributions in support of large-scale technology demonstration projects in the aerospace, defence, space and security sectors.
[xxv] The Federal Economic Development Agency for Southern Ontario, the Federal Economic Development Initiative for Northern Ontario, Canada Economic Development for Quebec Regions, and Western Economic Diversification Canada.
[xxvi] This total does not include clients who undertook one or more AIF projects under the IC sub-program and one or more BDP projects, which were all coded to the IBD or PG sub- programs.
[xxvii] From 2005-2006 to 2012-2013, average annual assistance approved by ACOA under the IC sub-program was approximately $26.2 million for AIF commercial projects, $25.6 million for AIF non-commercial projects, $25.6 million for BDP commercial projects and $5.0 million for BDP non-commercial projects, including Community Adjustment Fund (total $82.4 million).
[xxviii] The average ratings are summarized in Appendix E.
[xxix] Statistics Canada, CANSIM table 358- 0001. Gross domestic expenditures on research and development, by science type and by funder and performer sector (current prices).
[xxx] Note that “institutional” includes universities, colleges and research institutions; associations and public-sector organizations are excluded from this analysis.
[xxxi] The average ratings provided by clients are summarized in Appendix E.
[xxxii] Detailed ratings are provided in Appendix F.
[xxxiii] The average is calculated including those who generated no revenues.
[xxxiv] Pharmaceutical Research and Manufacturers of America, Biopharmaceutical Research Industry: 2013 Profile.
[xxxv] The financing does not involve the sale of a company's shares, and therefore does not dilute the ownership of existing shareholders.
[xxxvi] Detailed ratings are provided in Appendix E.
[xxxvii] Atlantic Canada Opportunities Agency, “Harper Government Unveils Major New Commitment to Supporting Innovation and Commercialization in Atlantic Canada,” July 16, 2014. http://www.acoa- apeca.gc.ca/eng/Agency/mediaroom/NewsReleases/Pages/4346.aspx.
[xxxviii] National Research Council of Canada, 2012-13 Departmental Performance Report – Supplementary Tables. http://www.nrc- cnrc.gc.ca/eng/reports/2012_2013/dpr_2013/dpr_table_01.html.
[xxxix] National Research Council of Canada, Annual Report 2012-2013. http://ww w.nrc- cnrc.gc.ca/eng/about/planning_reporting/annual/2012_2013/discussion_analysis.html#fsda2_2.
[xl] Natural Sciences and Engineering Research Council, 2012-13 Departmental Performance Report: Details on Transfer Payment Programs. http://www.nserc-crsng.gc.ca/NSERC-CRSNG/Reports-Rapports/plans- plans_eng.asp.
[xli] Natural Sciences and Engineering Research Council: Idea to Innovation Grants. http://www.nserc-crsng.gc.ca/Professors-Professeurs/RPP- PP/I2I-INNOV_eng.asp.
[xlii] Natural Sciences and Engineering Research Council: Collaborative Research and Development Grants. http://www.nserc-crsng.gc.ca/Professors- Professeurs/RPP-PP/CRD-RDC_eng.asp.
[xliii] Canadian Institutes of Health Research, 2012-2013 Departmental Performance Report – Supplementary Tables. http://www.cihr-irsc.gc.ca/e/47335.html.
[xliv] Social Sciences and Humanities Research Council, Funding by Region, Province and Institution for 2012-2013.
[xlv] Networks of Centres of Excellence of Canada: Funded Networks and Centres. http://www.nce-rce.gc.ca/NetworksCentres- CentresReseaux/Index_eng.asp.
[xlvi] Canada Foundation for Innovation: Projects Funded. http://www.innovation.ca/en/OurInve stments/Projectsfunded.
[xlvii] Business Development Bank of Canada, 2013 Annual Report. http://w ww.bdc.ca/EN/about/corporate_governance/financial_results/Pages/default.aspx.
[xlviii] Canada Revenue Agency: Who can claim SR&ED tax incentives and what are the benefits? http://www.cra-arc.gc.ca/txcrdt/sred- rsde/clmng/whcnclmsrd-eng.html.
[xlix] Industry Canada: TDP vs. SADI. http://www.ic.gc.ca/eic/site/ito- oti.nsf/eng/h_00860.html.
[l] Mitacs. http://www.mitacs.ca/.
[li] Government of Canada, Venture Capital Action Plan. http://www.fin.gc.ca/vcap-pacr/index-eng.asp.
[lii] Government of Canada, National Shipbuilding Procurement Strategy. http://www.tpsgc-pwgsc.gc.ca/app-acq/sam-mps/snacn-nsps-eng.html.
[liii] Sustainable Development Technology Canada. http://www.sdtc.ca/index.php? page=home.
[liv] Innovacorp, 2012-2013 Accountability Report. https://innovacorp.ca/about-us/corporate-publications.
[lv] New Brunswick Innovation Foundation. http://nbif.ca/en/about_nbif/.
[lvi] Nova Scotia Department of Economic and Rural Development and Tourism: Innovation and Learning. http://www.novascotia.ca/econ/ial/.
[lvii] Innovation PEI: Programs and Services. http://www.innovationpei.com/Programs-Services.
[lviii] Innovation PEI, 2012-2013 Annual Report. http://www.innovationpei.com/resources.
[lix] Research & Development Corporation of Newfoundland and Labrador, 2012-2013 Annual Report. http://www.rdc.org/%5C/about/index.htm.
[lx] Research & Development Corporation of Newfoundland and Labrador. http://www.rdc.org/%5C/funding/index.htm.
[lxi] New Brunswick Department of Economic Development. http://www2.gnb.ca/co ntent/gnb/en/departments/economic_development.html.
[lxii] Regional Development Corporation. http://www2.gnb.ca/co ntent/gnb/en/departments/regional_development.html.
[lxiii] RPC. http://www.rpc.ca/english/index.html.
[lxiv] Springboard. http://springboardatlantic.ca/about-us/.
[lxv] BioNova. http://bionova.ca/about/.
[lxvi] BioNB. http://bionb.org/en.
[lxvii] The Next Phase. http://thenextphase.ca/.
[lxviii] Propel ICT/Launch 36. http://launch36.ca/.
[lxix] Volta – Halifax Startup House. http://www.voltaeffect.com/.
[lxx] Venn Innovation Inc. http://venncentre.ca/.
[lxxi] Wallace McCain Institute. http://www.wallacemccaininstitute.com/.
[lxxii] Pond-Deshpande Centre for Innovation & Entrepreneurship. http://www.ponddeshpande.ca/.
[lxxiii] Genesis Centre. http://www.genesis.mun.ca/GenesisCentre/.
Page details
- Date modified: