This evaluation provides the Atlantic Canada Opportunities Agency (ACOA) management with systematic, neutral evidence on the relevance and performance of its Innovation and Commercialization sub-program. It builds on a recent internal review of the programming and supports ongoing decision making and monitoring of this critical element of ACOA’s portfolio.

The evaluation was designed and led by ACOA’s Evaluation Unit, and the study used a hybrid team approach involving ACOA’s internal evaluators and external consultants from Ference Weicker & Company. We thank the consulting team for their close collaboration, completion of primary research, synthesis, integration and reporting. We also thank the members of the evaluation advisory committee for their advice and support throughout the evaluation process. Their involvement and assistance helped ensure that the study and its recommendations are both relevant and useful. Of special note are the gracious contributions of three external advisory committee members and one external advisor: Rebecca Hart, Evaluation Officer, National Research Council of Canada; Dr. Jacquelyn Thayer Scott, Professor of Organizational Management and Public Administration, Cape Breton University and Chair, Board of Directors, Innovacorp; Peter Crooks, Executive Director, Canada’s Smartest Kitchen, Holland College; and Dr. Harald Bathelt, Canada Research Chair in Innovation and Governance, University of Toronto (external advisor).We are also grateful to the many ACOA staff members, including members of the Evaluation Unit, ACOA clients and external key informants who provided their time and essential knowledge in support of this study.

Overall, these contributions were instrumental in building our understanding and identifying best practices and opportunities for continuous improvement for ACOA’s Innovation and Commercialization programming.

Courtney Amo
Director, Evaluation and Risk Directorate (Head of Evaluation)
Atlantic Canada Opportunities Agency

Executive summary

The evaluation focuses on Innovation and Commercialization (IC) sub-program activities approved between 2005-2006 and 2012-2013. During this period, $652.5 million in ACOA assistance was approved for 1,131 IC projects. Operations and maintenance expenditures (O&M) totalled $56.2 million.

The IC sub-program is part of the Atlantic Canada Opportunities Agency’s (ACOA) Enterprise Development (ED) Program and is delivered through two grant and contribution (G&C) programs: the Atlantic Innovation Fund (AIF) and the Business Development Program (BDP). The AIF and BDP target both commercial and non-commercial entities (including research institutions, universities and other not-for-profit organizations). The AIF focuses on increasing research, development and innovation in Atlantic Canada and, for the period under review, was delivered through an Agency-wide competitive process. The BDP is delivered regionally and has a broader scope, encompassing a variety of eligible activities that support the growth and competitiveness of small and medium-sized enterprises (SMEs). ACOA recently revised its IC programming by launching a new BDP commercialization funding element, streamlining the intake of applications, changing the AIF process to a continuous intake system, and lowering the minimum funding threshold to better enable the AIF to meet business needs.

The evaluation of the IC sub-program is expected to contribute to learning, future decision making, and accountability (meeting the Treasury Board Policy on Evaluation requirement that all direct program spending be evaluated every five years). The evaluation employed multiple lines of evidence, including surveys of 277 IC clients, 16 case studies involving 52 projects, interviews with 60 key informants, a document and literature review, a review of project and operational data, the development of regional profiles and an economic impact analysis.

Conclusions

Relevance

There is a strong, continued need for the IC sub-program given the importance of innovation and commercialization to regional competitiveness, the need to build on the progress achieved in recent years, and the fit between the IC sub-program and the needs of clients. Innovation and commercialization are major drivers of economic competiveness. Atlantic Canada, and more specifically the IC sub-program, has made progress in strengthening commercialization capacity and increasing investment in research and development (R&D) over the past decade, yet the region continues to lag behind other regions of Canada on key innovation and commercialization indicators (e.g. level of investment in R&D, level of R&D performed by the private sector, access to venture capital and highly qualified personnel, proportion of federal research grant funding awarded, etc.). The IC sub-program responds to this need by providing support for innovation and commercialization projects that meet the needs of clients and would otherwise not proceed as planned.
In response to the results of an internal review, ACOA has recently made changes to better align the IC sub-program with client needs and evolving federal government priorities. These changes may impact demand for funding, types of clients and projects, results and efficiency.
The IC sub-program tends to complement rather than duplicate innovation and commercialization programming delivered by other organizations as well as other ACOA sub- programs. The IC sub-program is differentiated from other similar programs in its focus on commercialization, level of financial assistance provided, support for riskier projects and clients, and regional delivery model. To minimize the potential for overlap and leverage support for clients, ACOA plays a key leadership role in the region and coordinates activities with other programs. The IC sub- program also works closely with other ACOA programming in supporting clients along a continuum of development.
The IC sub-program is consistent with the strategic priorities, roles and responsibilities of the federal government and ACOA as well as strategies and approaches implemented by federal organizations in other regions of Canada and other jurisdictions. The IC sub-program aligns well with federal government priorities, particularly as outlined in the Science and Technology Strategy 2007, the Budget Plan 2014, speeches from the Throne and the 2011 Innovation Canada: A Call to Action (commonly known as the Jenkins Report), as well as with the strategic priorities, roles and responsibilities of ACOA. The focus of ACOA’s IC sub-program is consistent with approaches implemented by other federal government organizations (e.g. the high priority placed on innovation and commercialization, an emphasis on market “pull” innovation policy, and use of a broad definition of innovation).

Performance - effectiveness

The IC sub-program has resulted in extensive direct and leveraged investments in innovation and commercialization projects that, in turn, have generated a wide range of impacts in the immediate and intermediate term. Over the longer-term, the sub-program has expanded innovation and commercialization capacity and contributed to the growth and competitiveness of SMEs in Atlantic Canada. The $80 million in project funding approved annually by ACOA was leveraged with $1.32 in funding from other sources for every dollar contributed by ACOA and stimulated approximately $131 million in applied R&D expenditures, $57 million in commercialization expenditures, project employment for over 1,400 people, the development of 547 partnerships and 405 new technologies, products or processes, the adaptation of 370 technologies, and the commercialization of 521 technologies, products or processes that, as of the latest fiscal year, were generating $74 million in direct and spinoff revenues and 958 jobs. These results are understated in that the impacts will continue to grow as more projects reach the commercialization stage and revenues and employment from commercialization increase over time. The projects have also led to new patents and technology disclosures, further investment in R&D, cost savings and the attraction of new investment. Over the longer term, ACOA funding has supported the establishment and further development of various industrial clusters in Atlantic Canada, including aerospace and defence, life sciences and biotechnology, ocean technologies, information and communications technology, and energy (particularly renewable energy) sectors. The economic analysis, using the Conference Board of Canada’s (CBoC) econometric model, estimates that the IC sub-program generated an increase in gross domestic product (GDP) of $5.49 for every project dollar expended by ACOA over the 2007-2008 to 2012-2013 period.
The progress made to date has been influenced by the characteristics of the projects supported, the capabilities of client organizations, the design and delivery of the programming and economic conditions. The effective design and delivery of the sub-program has contributed to its success. Impacts vary across projects depending, in part, on focus (e.g. projects focused on commercialization require less time to generate economic impacts), the elapsed time since the project was completed (e.g. impacts increase over time), sector (e.g. certain sectors have shorter commercialization time frames than others), and the client’s technical and business skills. Factors that, at times, constrained project success include delays in project approval processes (i.e. all phases of application processing) or implementation and challenges in the economic environment. Some of the key lessons learned include the importance of being responsive to business needs in program delivery, supporting capacity development, offering a range of assistance to clients that can be tailored to their specific needs, and leveraging resources.
While ACOA staff have access to project monitoring and performance reporting information, there are specific challenges that need to be addressed. Data is used primarily for monitoring and information gathering, managing the portfolio, adjusting programming and Agency reporting. Challenges associated with performance measurement include difficulties in rolling up outcomes, the time required to see impacts, inherent difficulties in measuring innovation and commercialization, and outcome information not being entered into the system.

Performance – efficiency and economy

The existing governance structure, delivery model and funding mechanisms appear to support the efficient and economical delivery of the programming (O&M costs average about 8% of total program expenditures). Various factors contribute to the IC sub-program efficiency, including the regional governance structure (e.g. allows regions to tailor strategies and focus on needs), the client-centric delivery approach (e.g. offering a range of support mechanisms), the optimized use of resources (e.g. leveraging and repayability), and the collection and use of high-level performance information (e.g. using dashboards) to monitor portfolio performance and make investment decisions. Further improvements are expected as a result of recent changes to the programming. There may be opportunities to further improve efficiency and economy by sharing tools, processes, promising practices and lessons learned across regions.

Recommendations

The major recommendations arising from the evaluation of the Innovation and Commercialization sub-program are as follows:

Carefully monitor the recent programming changes to assess the impact and determine the need for further changes or adjustments.
Based on the results of its recent review, the Agency renewed its commitment to IC and is strengthening its programming by moving to a continuous intake system and lowering the minimum funding threshold to better enable the AIF to meet the needs of business as well as by launching a new BDP commercialization funding element. Some changes have also been made to the delivery structure. With these changes, the programming is expected to better operate at the speed of business and place a greater emphasis on commercialization (without excluding targeted investments in R&D capacity). The results should be carefully monitored to assess the impact of the changes on demand for funding, characteristics of clients and projects supported, approval times, risk, resulting impacts, and ongoing communication and sharing of information, both internally and externally, regarding client needs, recent developments, lessons learned and promising practices.
Revise the Performance Measurement Strategy (PMS) for the IC sub-program to reflect the recent program changes and improve the utility of the system for both performance reporting and management planning by integrating outcome data into monitoring systems.
Program logic models are being revised and the results will need to be reflected in a revised PMS. Difficulties in meaningfully rolling up project outcomes have meant that the impact information reported by clients has not been used extensively for either performance reporting or management planning. Limited use of the system has meant that a low priority is often placed on inputting impact or outcome information reported by clients into QAccess. To address these issues, ACOA should consider promising practices such as employing a mix of standard and unique indicators and incorporating a more client-focused approach rather than a more narrow project-focused approach to performance measurement.

Management has agreed with this evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the above recommendations, can be found in Appendix A.

1.0 Introduction

The Atlantic Canada Opportunities Agency (ACOA) promotes economic growth in Atlantic Canada by helping businesses become more competitive, innovative and productive, by working with diverse communities to develop and diversify local economies, and by championing the strengths of Atlantic Canada. ACOA’s mandate is “to enhance the growth of earned income and employment opportunities in Atlantic Canada.”

ACOA’s program alignment architecture (PAA) consists of four programs: Enterprise Development; Community Development; Policy, Advocacy and Coordination; and Internal Services. Enterprise Development consists of three sub-programs: Innovation and Commercialization; Productivity and Growth; and International Business Development. The IC sub-program is delivered primarily through two G&C programs: the Atlantic Innovation Fund (AIF) and the Business Development Program (BDP). In 2012-2013, spending under the IC sub-program accounted for $89.7 million of the $179.9 million dedicated to Enterprise Development.[i] The budget, sub-programs and performance indicators for each program as well as for ACOA overall are summarized in Appendix B.

1.1 Evaluation overview

The evaluation of the IC sub-program is expected to contribute to learning, future decision making and accountability (meeting the Treasury Board Policy on Evaluation requirement that all direct program spending be evaluated every five years). The evaluation covers the areas of program relevance (i.e. continued need for the program, alignment with government priorities, and alignment with federal roles and responsibilities) and performance (i.e. effectiveness, efficiency and economy). The scope of this evaluation includes IC activities approved between 2005-2006 and 2012-2013. This scope overlaps the scope of the previous Innovation evaluation (2010) to enable the evaluation to capture data on the longer-term results and impacts of the sub-program. The issues and questions addressed in this evaluation are provided in Appendix C.

Section 2 describes the evaluation methodology, limitations and mitigation strategies, section 3 provides an overview of the IC sub-program, sections 4, 5 and 6 summarize the major findings related to relevance, effectiveness, efficiency and economy, and section 7 presents the major conclusions and recommendations.

1.2 Evaluation approach and methodology

This evaluation used a hybrid team approach (internal evaluators and external consultants) in implementing a mixed method research design involving multiple lines of evidence. The evaluation complemented a formal internal review of innovation and commercialization activities recently undertaken by ACOA, utilizing some of the information collected and focusing research primarily on issues not covered as part of the review. The major lines of evidence for the evaluation include:

Document and Literature Review: A comprehensive review, focused primarily on issues related to relevance, was undertaken of internal and external documents related to the sub-program, funding programs, innovation and commercialization, federal policies and strategies, and previous evaluations. In addition, a review of similar activities in other jurisdictions was undertaken to identify alternative modes of delivery, possible lessons learned and best practices.
Review of Project and Operational Data: QAccess data were used to develop a detailed statistical profile of funded projects, client organizations, partnerships and the interrelationship between the three Enterprise Development sub-programs. In addition, operational data regarding resource allocations was reviewed and used in assessing efficiency and economy.
Regional Profiles: The results of the document and literature review, project data and operational data analysis were used in developing regional delivery profiles that describe how the sub-program is structured and delivered in each of the regional offices.
Client Survey: A web-based and telephone survey was undertaken, targeting 530 of the 762 AIF and BDP clients funded through the IC sub-program between 2005-2006 and 2012-2013. A maximum of 489 companies were contacted by e-mail or telephone. Each client was asked about one project, selected by the ACOA Evaluation Team on the basis of materiality (i.e. funding provided) and year of funding, with preference given to projects undertaken in the earlier years covered by the evaluation. Of these 489 clients, 277 responded, which represents 52% of the clients in the sample population, 24% of the projects in the total population of IC projects, and 36% of the unique clients funded from 2005-2006 to 2012-2013. At a confidence level of 95%, the sample of 277 respondents achieves a margin of error of about ±4.0%.

Table 1: Response Rate, by Program and Type of Client

Program	Type	Surveyed	Project Sample	Response
BDP	Commercial	202	396	51%
	Non-commercial	28	54	52%
	Sub-total	230	450	51%
AIF	Commercial	24	49	49%
	Non-commercial	23	31	74%
	Sub-total	47	80	59%
Total		277	530	52%

Source: ACOA QAccess project management system, data extracted September 2013.

Case Studies: Sixteen case studies, involving 52 projects (18 AIF- and 34 BDP-funded projects), were selected for in-depth review based on criteria such as region, sector, client characteristics and availability of performance data. The case studies illustrate the nature of the impacts, the lessons learned, contributing and constraining factors, and the interrelationship between projects assisted by ACOA. The case studies involved a review of documents and data (e.g. proposals, tracking forms, progress reports, final reports and publicly available information) and interviews with 53 representatives from proponent organizations, ACOA, other sources of project funding, and others involved in or impacted by the project.
Key Informant Interviews: Semi-structured interviews were conducted with 60 key informants, including 27 representatives of ACOA (as part of 16 group interviews), 17 representatives of other government departments and partners, 12 stakeholders and experts, and 4 representatives of similar programs delivered by other regional development agencies (RDAs) of the Government of Canada. The primary focus was to obtain input on the need for the programming, its relationship to other programs, factors that contribute to and constrain the achievement of the intended outcomes, lessons learned, best practices and opportunities for improvement.
Economic Impact Analysis: The results of the client survey and project data were analyzed to provide input into an economic impact analysis that uses the Conference Board of Canada’s dynamic econometric model of the four Atlantic provinces. The analysis estimated the direct, indirect and induced economic impacts of IC sub-program activities.

1.3 Evaluation challenges and mitigation measures

The major challenges associated with the evaluation and the strategies employed to mitigate them are described below.

Not enough time has elapsed for the impacts of the projects to be fully realized. Only 70% of the 277 projects surveyed were completed at the time of evaluation. Projects that have been completed (particularly certain types of projects such as AIF projects or projects in the health sector) may require additional time post-completion to realize their full impacts. To mitigate this challenge, the selection of projects for the client survey and case studies placed a higher priority on those undertaken in the earlier years covered by the evaluation. In addition, the results of the survey were analyzed to illustrate the growth in impacts over time. Given the sample size, the estimates are considered reliable, although the reported results are understated as impacts are expected to continue to grow.
The potential for non-response error: Factors that contributed to the potential for non-response error included difficulties in reaching some respondents, the completeness and accuracy of some contact information, and the length of time since the projects were completed (which may impact the availability of clients as well as their willingness and ability to respond). To mitigate these potential effects, we followed-up with target respondents multiple times, offered them the option of responding through multiple modes and, where necessary, identified alternative respondents within the same organization to interview or survey. We also confirmed that the characteristics of the survey respondents were broadly representative of the target population.
Potential for respondent bias: The majority of those surveyed or interviewed were involved with or beneficiaries of the sub-program. To mitigate this, the purpose of the survey or interview, the design and methodology, and strict confidentiality of responses were communicated clearly to participants. In addition, the sample of key informants included federal, provincial and non-profit representatives who are not directly involved in the sub-program, and the survey and interview results were triangulated with data obtained through other lines of evidence.

Given these mitigation measures, the limitations of the study were adequately addressed and the results of the IC sub-program evaluation are deemed to be reliable and valid.

2.0 Profile of the innovation and commercialization sub-program

2.1 Sub-program profile

Projects coded to the IC sub-program are funded through the AIF and the BDP. Launched in 2001, the AIF aims to strengthen the region’s level of activity in and capacity for innovation, R&D, and commercialization as well as to maximize benefits from national R&D programs.

Launched in 1995, the BDP has a very broad scope, encompassing a variety of eligible recipients and activities. The objectives of the BDP are to improve the growth and competitiveness of Atlantic SMEs, provide for dynamic and sustainable communities in Atlantic Canada, and provide for policies and programs that strengthen the Atlantic economy.

The AIF and BDP target both commercial and non-commercial entities (including research institutions, universities, and other not-for-profit organizations such as industry associations). Indirect stakeholders of the IC sub-program include industry groups and associations, federal science-based departments, national R&D programs and provincial governments.

2.2 Sub-program logic model

As indicated in the logic model in Appendix D, the activities and outputs of the IC sub-program consist primarily of financial assistance for innovation and commercialization projects. In addition, ACOA staff provide in-kind assistance to clients in the form of help in navigating ACOA programming, referrals to other programs, and one-on-one assistance with business planning and strategies.

In the immediate term, the financial assistance is intended to increase applied R&D activity, capacity for applied R&D, adaptation of technology, and capacity for commercialization; to establish networks, partnerships, and alliances; and to attract, develop and retain R&D staff. In the intermediate term, it is expected to result in new or improved technologies, products, processes and/or services; benefits from increased applied research capacity; further investment in R&D; and commercialization of research. In the long term, the sub-program is intended to strengthen the capacity for innovation and commercialization in Atlantic Canada, which will contribute to the intended outcome for the Enterprise Development Program (“improved growth and competitiveness of Atlantic SMEs”) and the strategic outcome for the Agency overall (“a competitive Atlantic Canadian economy”).

2.3 Sub-program expenditures

Grants and contributions

From 2005-2006 to 2012-2013, $653 million in funding was approved, involving 1,131 projects and 762 unique clients. Of this funding, $414 million (64%) was approved through the AIF and $238 million (36%) was approved through the BDP. AIF funding for commercial clients consists of interest-free, unsecured, repayable or conditionally repayable contributions, while the funding for non-commercial clients consists of non-repayable contributions. AIF funding per project ranged from about $300,000 to $10 million in value, averaging about $2.3 million. BDP assistance for commercial clients consists of repayable or conditionally repayable interest-free, unsecured contributions, while assistance for non-commercial projects is in the form of non-repayable contributions or grants. BDP funding per project ranged from nearly $5,000 to $5 million in value, averaging $255,000.

Table 2: Profile of ACOA Assistance, by Program and Type of Client, 2005-2006 to 2012-2013

Program/Client	Approved Projects	ACOA Assistance ($M)	Percentage of Projects That Are:
Program/Client	Approved Projects	ACOA Assistance ($M)	Repayable	Conditional		Non-Repayable	Grant	Total
BDP
Commercial	790	$198.0	76%	22%	2%		--	100%
Non-comm.	162	$40.1	0%	0%	90%		10%	100%
Total BDP	952	$238.1	63%	19%	16%		2%	100%
AIF
Commercial	90	$209.5	3%	96%	1%		0%	100%
Non-comm.	89	$204.9	0%	0%	100%		0%	100%
Total AIF	179	$414.4	2%	48%	50%		0%	100%
Total	1,131	$652.5	53%	23%	23%		1%	100%

Source: ACOA QAccess project management system, data extracted September 2013.

The budget allocation for the BDP is guided by regional allocations. Adjustments may be made mid-year, taking into account spending to date and expected expenditures, including any special initiatives.

The AIF does not operate with a regional allocation and investments are based on a competitive process. The AIF was originally approved by the Treasury Board as a $300 million, five-year initiative that was administered in two large rounds (approximately $150 million per round). The second envelope (2006-2009) was administered in four smaller rounds (approximately $60 million per round). In its 2010 Budget, the Government of Canada made the AIF a permanent, ongoing program.

In the Economic Action Plan 2014, the Government reaffirmed its commitment to increasing business innovation, productivity and competitiveness in Atlantic Canada. As part of this commitment, ACOA will contribute $450 million over the next five years to innovation projects through the AIF and BDP[ii]

Details regarding the types of projects supported by the IC sub-program are available in section 5.1.

Operations and maintenance expenditures

Operations and maintenance expenditures related to the IC programming totalled $56 million from 2005-2006 to 2012-2013, including $11 million in operating expenses and $45 million in salaries as shown below.

Table 3: Operations and maintenance expenditures, 2005-2006 to 2012- 2013

Fiscal Year	Operating Expenditures	Salary Expenditures	Total
2005-2006	$1,645,672	$4,820,876	$6,466,548
2006-2007	$1,411,099	$5,351,742	$6,762,841
2007-2008	$1,745,916	$5,380,965	$7,126,881
2008-2009	$1,600,930	$5,373,912	$6,974,842
2009-2010	$1,605,532	$6,677,926	$8,283,458
2010-2011	$1,558,766	$6,018,032	$7,576,798
2011-2012	$980,741	$5,302,968	$6,283,709
2012-2013	$722,901	$6,049,787	$6,772,688
Grand Total	$11,271,557	$44,976,208	$56,247,765

Source: QAccess ACOA GX financial management system, data extracted May 2014.

As shown in the table below, from 2007-2008 to 2012-2013, full-time equivalents (FTEs) estimated to be dedicated to IC programming ranged from 66 to 80, representing about 30% of all FTEs dedicated to Enterprise Development (ED) and about 10% of total ACOA FTEs.[iii] FTEs reflect an estimated proportion of staff time dedicated to IC programming by program officers/economic development officers/policy analysts, payment officers, managers, directors, administrative staff and upper-level management. Delivery structures and types of staff dedicated to IC programming vary by region.

Table 4: FTEs dedicated to innovation and commercialization, 2007-2008 to 2012-2013

Fiscal Year	IC FTEs	Total ED FTEs	Total ACOA FTEs
2007-2008	69.4	441	726
2008-2009	66.2	424	723
2009-2010	79.9	212	718
2010-2011	74.3	215	714
2011-2012	70.3	203	683
2012-2013	69.5	181	588

Sources: IC FTE data extracted from GX data. Total ACOA FTEs data extracted from Departmental Performance Reports for fiscal years 2007-2008 to 2012-2013.

2.4 Governance and management

Organizational structure

IC sub-program activities are overseen by the Director General (DG) of Enterprise Development located in ACOA’s head office, together with regional directors of Enterprise Development. The DG Enterprise Development reports directly to the Vice-President, Policy and Programs, while the regional directors of Enterprise Development report to the DG of Operations within their respective region. Major programming decisions are made by ACOA’s Executive Committee, which includes ACOA regional vice-presidents and related senior executives as identified by the President to ensure the appropriate representation of expertise and experience.

ACOA’s Head Office is located in Moncton, N.B. Regional offices are located in Halifax, N.S.; Charlottetown, P.E.I.; St. John’s, N.L.; and Fredericton, N.B. Some IC programming had also been delivered through the former Enterprise Cape Breton Corporation, whose economic and community development activities, including the associated budget, were recently transitioned over to ACOA. Each regional office is headed by a vice-president and is responsible for program delivery within its respective province and for supporting Head Office in the development of program policies. Some regions have satellite offices that support the delivery of ACOA programming (e.g. eight in New Brunswick, six in Newfoundland and Labrador, and nine in Nova Scotia).[iv]

Delivery processes

Prior to recent changes[v]AIF projects were approved through an annual competitive process. While project proposals were normally accepted through a request for letters of intent and project proposal process, projects could be accepted on an exceptional basis under the strategic initiatives element of the program, which involved a separate application-based process. The AIF is coordinated and managed through the AIF Secretariat established at Head Office. The Secretariat’s main objective is to guide managers in delivering the AIF program and to coordinate the intake of project proposals. The AIF Secretariat coordinates the scientific review process, specifically the involvement of other federal departments and agencies (e.g. National Research Council Canada). For additional expertise assistance, account managers draw from federal partners or from external consultants. Sector Teams provide advice and perform a challenge function on project proposals. The AIF Secretariat also liaises with the AIF Advisory Board and ensures that they have sufficient information to perform their duties. The AIF Advisory Board is an independent body established to evaluate and recommend projects for funding to the minister and president.

In contrast, the BDP follows a continuous, demand-driven intake process with no specific funding decision rounds. Authority is delegated to various levels depending on the regional office and type of project, and is also established as per the Agency’s financial signing authority delegation instrument. Across the regions, the level of signing authority for a BDP contribution ranges from no signing authority to up to $100,000 for account managers, up to $300,000 for managers, up to $500,000 for directors, director generals of operations, and regional vice-presidents. Above that level, the president can approve contributions up to $ 10 million, however direction from the Minister is sought for all contributions between $1 million up to $10 million.

The regions also differ in their approach to project development. Some regions have committees to review projects (e.g. New Brunswick has a Projects Under Development Committee that meets monthly) and some regions have teams that seek out projects in specific sectors (e.g. Newfoundland and Labrador has Opportunities-Based Teams that focus on developing specific economic opportunities for industries, regions and businesses such as supplier development to the oil and gas industry), and some regions (e.g. Nova Scotia) work closely with partners and the private sector to develop clients in high-growth sectors.

3.0 Findings: relevance

This section summarizes findings with respect to need for the sub-program, relationship with other programming, and alignment with ACOA and federal government priorities, roles and responsibilities.

3.1 Continued need for the programming

The results of the evaluation demonstrate a strong, continued need for the IC sub-program given the importance of innovation and commercialization to regional competitiveness, the need to build on the progress achieved in recent years, and the fit between the IC sub- program and the needs of clients.

Continued need

Innovation and commercialization are major drivers of business productivity and the economic competiveness of a region.

According to the literature, there is a strong link between innovation, productivity and, ultimately, the economic growth and competitiveness of a region. The Organisation for Economic Co- operation and Development (OECD) Oslo Manual states: “It has been long understood that the generation, exploitation and diffusion of knowledge are fundamental to economic growth, development and the well-being of nations.” The Manual defines innovation as “the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new organizational method in business practices, workplace organization or external relations.”[vi]

Innovation leads to enhanced growth and competitiveness by making businesses more productive. The World Economic Forum Global Competitiveness Report defines competitiveness as “the set of institutions, policies and factors that determine the level of productivity of a country.” The report explains that there have been many productivity gains throughout history attributable to innovation and that these gains require an environment conducive to innovative activity supported by both the public and private sectors (e.g. sufficient investment in R&D, particularly by the private sector, the presence of high quality scientific research institutions, extensive collaboration between universities and industry, the protection of intellectual property, high levels of competition, and access to venture capital and financing). The report further states that investment in these activities is critical to sustainable growth.[vii]

Canadian policy studies have also identified the link between innovation, productivity, competitiveness and growth, particularly with respect to the commercialization of innovations. Innovation Canada: A Call to Action explains that productivity growth drives increases in average per capita income and business competitiveness, and that productivity growth is primarily the result of innovation. The report emphasizes that innovation diffusion and adaptation are critical to productivity growth [viii],which is achieved through successful commercialization of innovative technologies, products, processes or services.

Over the past decade, considerable progress has been made in strengthening the capacity for innovation and commercialization and increasing investment in R&D in Atlantic Canada. Some areas of progress highlighted in recent reports include:

Increased recognition of the importance of research and innovation policies and programs in promoting economic prosperity, as reflected in new programming and plans (e.g. key initiative of the Council of Atlantic Premiers) and greater interprovincial and federal- provincial collaboration.
Further development of research capacity. Sixteen universities and five college systems perform R&D, train students and researchers, and create spinoff businesses. Eight Atlantic Canada universities were ranked among Canada’s 50 top research universities in 2011 in sponsored research income.[ix] There is a wide range of public and non- governmental research institutes.
Increased collaboration between academia/research institutions and the private sector, catalyzed in part through an increased emphasis on collaboration in funding provided under various government programs, including ACOA’s. The establishment of Springboard Atlantic (a technology transfer network of 13 universities and five colleges that supports commercialization of university research in Atlantic Canada) in 2004 has strengthened university-industry liaison and technology transfer capacity, among other areas.
Further development of other resources, networks and ecosystems that facilitate commercialization by providing financing, training and mentoring. Examples include the First Angel Network Association, which delivers education and networking opportunities related to investment readiness and commercialization, Learnsphere, which focuses on skills development, and industry associations, which facilitate partnerships.
Increased investment in R&D. According to Statistics Canada data, business enterprise R&D (BERD) spending grew at a higher rate in Atlantic Canada (4.7%) compared to the rest of the country (1.1%) from 2001 to 2011.[x]

However, the region continues to lag other parts of Canada and other countries on key innovation and commercialization indicators and faces significant barriers, highlighting the continued need for innovation and commercialization programming. Atlantic Canada continues to lag behind Canada on key innovation measures and productivity, which hinders the ability of its firms to grow and compete internationally. Relative to other Canadian provinces, the region experiences:

Low levels of BERD and R&D performed by the private sector. Although 7% of Canada’s population resided in Atlantic Canada in 2011, the region accounted for less than 2% of national BERD. Twenty percent of the R&D activities in Atlantic Canada are performed by the business sector, as compared to 60% nationally. There is empirical evidence that R&D performed by business, rather than by universities and governments, contributes most directly to productivity growth.
Lower productivity and corresponding level of investment in machinery and equipment. Atlantic Canada’s labour productivity in 2011 was equal to 84.6% of the Canadian average.
More limited access to angel and venture capital financing. Though access to growth capital has increased in recent years (from 2% in 2008), Atlantic Canada received only 4.6% of the venture capital invested in Canada in 2011.
Lower rate of intellectual property creation. Only 2% of patents filed and granted in Canada are from Atlantic Canada.[xi]
More limited access to federal research grant funding. Even though the level of federal research grant funding awarded to Atlantic Canada institutions has increased substantially over the past 10 years, the proportion of funding awarded has not increased.[xii] Most of the increase in funding has been awarded to a few larger universities (e.g. Dalhousie, Memorial and New Brunswick), with smaller institutions continuing to face challenges competing for funds due to limited capacity.

Furthermore, Canada lags other countries on similar key innovation measures such as BERD as a percentage of GDP (1% versus 1.6% OECD average in 2011), business-sector labour productivity (75% of the level in the United States), collaboration between business and post-secondary sectors, and doctoral graduates in science and engineering (ranking 21^st among OECD countries in percentage of new degrees awarded in science and engineering). According to the CBoC, Canada ranks 13^th among 16 peer countries in terms of innovation, lagging particularly in terms of commercialization.

Significant barriers that contribute to the innovation and productivity gaps in the Atlantic region include a historical focus on primary resource industries, a small (regional) domestic market with a highly rural population, significant interprovincial and regional disparities, an aging population, a high level of outmigration, limited success in attracting and retaining immigrants and skilled workers (both technical and managerial), and a risk-averse business culture. Working in association with ACOA Enterprise Development and Community Development sub-programs, the IC sub-program targets these barriers by making strategic investments in projects that expand R&D capacity, build linkages between universities/research institutions and businesses, support the commercialization of research activities and encourage Atlantic SMEs to engage in innovation activities by sharing the financial risk and leveraging other resources from public- and private-sector financial partners.

Responsiveness to needs

IC sub-program funding facilitated the implementation of projects that met the needs of clients and that would otherwise not have proceeded as planned.

Most clients surveyed indicated that the assistance provided by ACOA met their needs, and most case study proponents were satisfied with the assistance received. Only a few of the clients surveyed indicated that ACOA did not meet their needs, noting the following: more funding was needed; greater flexibility was needed with respect to the completion date or how the funding was used; the funding should have been conditionally repayable or non- repayable; or there were delays in processing their application or claims.

ACOA support is seen as critical to project implementation, as only 4% of clients surveyed (0% of AIF and 5% of BDP clients) would have proceeded with the project as planned in the absence of ACOA support. Projects would most likely have been reduced in scope (39%), cancelled (35%), delayed (25%) and/or implemented over a longer period of time (24%). On average, clients estimated that there was a 35% likelihood that the projects would have proceeded in some form without ACOA support. In the absence of ACOA support, 18 percent of surveyed clients indicated that they would have approached another organization or program for assistance, such as the National Research Council, the provincial government, other federal government departments, private-sector investors and financial institutions.

According to the 2014 Innovation evaluation, the percentage of clients who indicated that the project would have been cancelled without ACOA funding was somewhat higher for AIF projects relative to the results reported in the 2010 evaluation (51% would not have proceeded, compared to 46% reported in 2010) and lower for the BDP projects (31% compared to the 46% reported in 2010).

To ensure continued relevance of the IC sub-program, ACOA has monitored the changing innovation and commercialization landscape and very recently adjusted its programming to better meet client needs and reflect evolving federal government priorities.

The results of the document and literature review illustrate that the IC sub-program is responsive to needs in that it strengthens factors that drive innovation and commercialization performance. The Agency commits significant funding to promote innovation and commercialization activities and places an emphasis on the development of SMEs and clusters in key strategic sectors. It facilitates responsiveness through its regional design and delivery model, which enables support to be tailored and adjusted over time to meet existing and emerging needs. Account managers play a critical front-line role in linking ACOA programs with client needs.

A majority of ACOA representatives believe that the Agency has access to the information it needs to adjust its programming and strategies in response to changes in the innovation and commercialization landscape. For example, the Agency recently conducted a formal internal review of innovation and commercialization activities. The review involved extensive consultations with key industry and institutional stakeholders across Atlantic Canada and resulted in adjustments to the IC programming (e.g. an increased focus on commercialization, a more streamlined application process and increased flexibility in types of eligible expenditures).

On an ongoing basis, ACOA tracks changes in the innovation and commercialization landscape on a less formal basis, relying on information obtained from federal partners, provincial partners and other key stakeholders (e.g. industry associations and start-up accelerators), through networks, events and forums (e.g. the Atlantic Venture Forum), clients, program staff, program evaluations and reviews, Statistics Canada and provincial government sources, and research reports. Some ACOA representatives noted that it is difficult to keep up with the pace of change in the innovation and commercialization landscape. Information gaps were identified with respect to broader client or systemic issues (e.g. access to angel investment, venture capital and other support services), data on specific industrial clusters, and the needs of specific areas or segments (e.g. the needs of rural areas).

3.2 Relationship to other programming

The IC sub-program is differentiated from other programs that promote innovation and commercialization in Atlantic Canada by its focus on commercialization, the level of financial assistance provided, its support for riskier projects and clients, and a regional delivery model.

A variety of programs delivered by the federal government, provincial governments and others promote innovation and commercialization in Atlantic Canada. While statistics are not available on investment in commercialization activities in the region, data are available on R&D investments. Statistics Canada data indicate that, in 2011, approximately $1.16 billion was invested in R&D in Atlantic Canada, of which the leading sources were the higher education sector ($429 million), the federal government ($336 million), the business sector ($287 million), the provincial governments ($50 million) and others ($60 million).[xiii]

The following figure draws from data found in Appendix E to provide an overview of leading sources of funding for innovation and commercialization in Atlantic Canada, indicating the average value of G&Cs awarded in Atlantic Canada (based on the most recent year’s data) and the relative position of the program along an innovation continuum from discovery to commercialization. As indicated, ACOA is the largest single source of program funding in the region. The Business Development Bank of Canada does not provide G&Cs but is an important source of funding for commercialization through its loan programs.

Figure 1: Overview of Leading Programs in Atlantic Canada [xiv]

(D)

The ACOA IC sub-program is viewed by most key informants and case study proponents as complementing rather than duplicating other programs and sources of financing, noting that ACOA occupies a niche that is not currently filled by other programs. In comparison to other programming, the ACOA IC sub- program differs in the following ways:

Relative emphasis on commercialization. Both key informants and case study representatives noted that the focus of ACOA’s programming has been shifting away from R&D capacity building toward a greater emphasis on commercialization. Other leading programs (e.g. NSERC and CIHR) tend to focus more heavily on earlier stage research.
Level of financial assistance available. Key informants, clients and case study representatives noted that ACOA’s IC sub-program complements other programs by offering higher levels of funding for innovation and commercialization projects.
The types of assistance provided and the nature of projects funded. The IC sub-program offers interest-free, unsecured, repayable, conditionally repayable or non-repayable contributions and supports a wider range of projects, including higher risk projects (e.g. projects in emerging sectors). Few projects would have proceeded as planned in the absence of ACOA funding.
The regional delivery model. ACOA has a strong staff presence in the regions (60 to 70 FTEs per year dedicated to IC activities delivered by 28 regional and satellite offices). Resources on the ground facilitate greater communication with clients; support the development and processing of applications; and enable the Agency to better understand local needs. Assistance can then be tailored specifically to meet client needs and account for regional context.

Clients confirmed that ACOA’s IC programming offers unique advantages over other existing programs. When clients were asked to rate how important various factors were in their decision to apply to ACOA for support rather than to another organization, the highest rated reasons were the level of funding available, eligibility requirements, timeliness of project approvals, and the definition of eligible expenditures. Responses did not differ significantly by client type (i.e. BDP versus AIF, commercial versus non-commercial).

To minimize the potential for overlap and to leverage support for clients, ACOA plays a key convening role in the region and coordinates activities with other programs.

Some key informants noted a few areas of overlap between the IC sub-program and other programs. It was noted that the AIF overlaps, to some extent, with other federal government funding programs that provide support for applied R&D at universities (e.g. the NSERC Idea to Innovation Grant, which provides funding to college and university faculty members to support R&D projects with recognized commercial potential) and for projects that promote partnerships between universities and SMEs (e.g. Networks of Centres of Excellence). Some overlap was also noted with certain provincial government programs such as the former Department of Innovation Business and Rural Development in Newfoundland [xv] (which offers contribution programs aimed at both businesses and universities) and Innovation PEI (which funds research students).

Most key informants agreed that the areas of overlap are minimal and readily addressed through communication and collaboration between ACOA and its partners. Key informants explained that ACOA coordinates its activities with those of other programs through direct communication with partners regarding support for particular projects, by convening with partners to review programs and coordinate strategies, and by leveraging partners’ scientific and sector expertise. It was noted that ACOA’s involvement in a project can help reduce financing risk and bring additional legitimacy to a project. ACOA’s regional offices also monitor changes in programming support and adapt their focus, as warranted, to fill specific gaps and complement available provincial and federal resources. A few key informants noted that, in some cases, various factors can constrain the level of communication, coordination and collaboration between ACOA and other organizations. These factors include differing organizational objectives and mandates (which can, for example, complicate reporting requirements when funding the same client), the limited involvement of some provincial governments in innovation and commercialization, and difficulties in coordinating activities with programs delivered through departments/agencies outside the region.

The IC sub-program works closely with other ACOA programming in providing direct and indirect support to clients along a continuum of development. Just over one half of clients received funding from another ACOA sub-program during the period covered by the evaluation.

The IC sub-program is viewed by key informants as complementary to other ACOA programming. For example, projects funded under the IC sub-program may result in the development and commercialization of new technologies. Funding provided under the Productivity and Growth (PG) sub-program can support the production and sale of these technologies. Funding under the International Business Development (IBD) sub-program can support the development and expansion of international markets for these technologies. The funding provided to commercial and non-commercial clients works together to support companies along the continuum (e.g. mixing direct assistance to businesses with indirect assistance through non-profit support organizations). ACOA representatives noted that the support provided by the three sub- programs is not necessarily linear but that the programs are complementary because they work toward the same ultimate goals and outcomes of enhanced competitiveness, productivity and growth.

Data from ACOA’s project and client management system (QAccess) indicate that clients are accessing complementary support from ACOA. Of clients funded for one or more IC projects, 48% received funding for IC projects only, 39% received funding for both IC and PG projects, 5% received funding for both IC and IBD projects, and 8% received funding for IC, PG and IBD projects. This complementarity will be further explored in ACOA’s evaluations of IBD and PG, which are currently under way.

3.3 Alignment with federal priorities, roles and responsibilities

Alignment with federal government priorities and ACOA priorities

The IC sub-program is well aligned with federal government priorities as outlined in various strategies, budget plans, and speeches from the Throne as well as with the strategic priorities, roles and responsibilities of ACOA. The primary objectives of the IC sub-program are to support the leveraging of funds toward innovation projects, encourage the establishment of partnerships for technology development and commercialization, and increase revenues resulting from commercialization. These objectives are consistent with the priorities of the federal government, particularly as stated in:

Canada’s Science and Technology Strategy 2007, [xvi] which states that the Government of Canada will foster a competitive and dynamic business environment and pursue public-private research and commercialization partnerships.
Budget Plan 2014, which has a specific focus on supporting innovation in Atlantic Canada “to ensure that they are responsive to the needs of business, including small and medium-sized enterprises, and that they effectively support the development and commercialization of new ideas, technologies, products and services that lead to strong economic growth.” [xvii]
Speeches from the Throne. The 2014 Speech from the Throne emphasizes that “our Government will continue making targeted investments in science and innovation chains from laboratory to market in order to position Canada as a leader in the knowledge economy.” [xviii] Similarly, the 2011 Speech from the Throne highlights that “our Government will continue to make targeted investments to promote and encourage research and development in Canada’s private sector and in our universities, colleges and polytechnics.” [xix}
Innovation Canada: A Call to Action [xx] emphasizes that priority should be given to fostering collaboration with key actors in the innovation ecosystem: industries, provinces, post-secondary institutions, civil society organizations and relevant investor communities. The scale of collaborative projects should change from investigator-led “idea push” projects to large-scale, industry-facing, demand-driven and outcome-oriented projects.

The objectives are also consistent with ACOA’s “programs and services on initiatives that encourage Atlantic Canadian businesses to become more innovative, productive and competitive in the global marketplace” priority as well as the “competitive Atlantic Canadian Economy” strategic outcome in its 2013-2014 Report on Plans and Priorities, as well as ACOA’s renewed focus on rural SMEs.

ACOA is aware of the priorities and strategies of its partners (federal, provincial, municipal) and fosters alignment where appropriate. Representatives of other government agencies and programs agreed that the sub-program also aligns fairly well with the priorities and strategies of their organizations and those of provincial and regional partners. ACOA’s regional offices work closely with partners to coordinate innovation and commercialization programming. For example, the regional office in Fredericton, N.B., engages the province to examine the roles, responsibilities and structure of economic development partners to increase their impact and efficiency. The Newfoundland and Labrador office places a priority on further developing partnerships/alliances with IC stakeholders. The Prince Edward Island regional office works closely with the provincial government on ED programming and provincial strategies. The provincial government of Nova Scotia actively promotes innovation (e.g. hosting the province’s first Innovation Summit in April 2013).

Alignment with federal government roles and responsibilities

The focus, priorities and roles of ACOA’s IC sub-program are consistent with approaches implemented by federal government organizations in other regions of Canada and internationally.

The IC sub-program is well aligned with federal roles and responsibilities. Although the federal government has long been involved in regional economic development, it was only with the passing of the Constitution Act, 1982 [xxi] that, under section 36, the Government of Canada and the provincial governments committed “to furthering economic development to reduce disparity in opportunities.” The 1988 ACOA Act established ACOA as the primary federal department responsible for economic development efforts in Atlantic Canada [xxii].

ACOA’s roles and responsibilities continue to be relevant even in light of recent developments in the innovation landscape: Jenkins report; the Government of Canada’s intention to release an updated Science, Technology and Innovation Strategy [xxiii]; the transformation of the National Research Council Canada (NRC); and the introduction of the programs such as the Venture Capital Action Plan, the Canada Accelerator and Incubator Program, and the Technology Demonstration Program. [xxiv] Some of the major policy shifts in Canada include the move to market “pull” innovation policy; the adoption of a broader definition of innovation (e.g. to include process and organizational innovation); and an increased focus on SMEs as innovation drivers. Key informants agreed that the sub-program reflects these developments.

The sub-program also aligns well with trends among other federal agencies inside and outside of Canada. Representatives from other Canadian federal RDAs [xxv] described innovation and commercialization as a central or increasingly important priority for their organizations. These agencies are focusing on addressing commercialization gaps, developing strategic partnerships between SMEs and research organizations, and enhancing the productivity of SMEs. Governments in other countries are also placing an increased focus on innovation support: Australia is undergoing a consolidation of innovation programs and is providing targeted support for commercialization (e.g. Commercialization Australia); the United Kingdom supports innovation through public procurement (e.g. Technology Strategy Board), among other strategies; Finland has a complex system of support for innovation, with a particular focus on supporting skills and talent development; and Northern Scotland development organizations such as the Highlands and Islands Enterprise promote innovation by promoting partnerships between businesses and academic institutions, among other mechanisms.

4.0 Findings: performance – effectiveness

This section summarizes the key findings regarding the outputs and outcomes of the IC sub-program, factors contributing to and constraining effectiveness, and performance measurement. Findings are presented at an aggregate level when no significant differences are observed between programs (AIF versus BDP) or client types (commercial versus non-commercial).

4.1 Investment in innovation and commercialization projects

The IC sub-program has facilitated extensive investment in projects, including funding provided by ACOA as well as funding leveraged from other sources.

During the evaluation period, from 2005-2006 to 2012-2013, ACOA approved over $650 million in financial assistance for 1,131 innovation and commercialization projects involving 762 unique clients (an average of 1.5 projects per client). For every dollar of funding provided by ACOA, $1.32 was provided by other sources, including funds contributed by the client organization (reported by 55% of projects surveyed), provincial government (38%), other federal government sources (29%), private-sector partners (26%), private equity (19%), commercial lenders (16%) and universities or research institutions (10%). The leverage rate exceeds the target rate of $1.05 established by ACOA but is lower than the leverage rate reported in 2010 ($1.44).

The results of the client survey, case studies and key informant interviews indicate that the presence of ACOA funding better enables clients to access and use funding from other sources. Clients estimated that, on average, only 34% of the funding from other sources would still have been invested in their project if they had not received assistance from ACOA. Case study representatives emphasized the role of ACOA assistance in fostering the sharing of project risk and better enabling projects to access private-sector financing. Key informants and case study representatives noted that project funding from ACOA and other public-sector partners is often mutually conditional.

The nature of the innovation and commercialization projects and the resulting impacts vary significantly depending upon the program under which they are funded (BDP or AIF) and the type of client (commercial or non-commercial).

The IC sub-program is best viewed as a series of four interrelated program areas (AIF commercial, AIF non-commercial, BDP commercial and BDP non-commercial) that work together to promote innovation and commercialization. While these program areas work jointly toward the intended outcomes of the sub-program, they vary significantly in terms of the types of clients served, projects supported, time frames, and nature of impacts generated.

Appendix G provides a profile of each program area in terms of the client and project characteristics. As indicated, AIF commercial and non-commercial projects each accounted for only 8% of the 1,131 projects approved but 32% and 31% respectively of the $653 million in funding approved by ACOA. In contrast, the BDP commercial clients accounted for 70% of projects but only 30% of ACOA funding, while BDP non-commercial projects accounted for 14% of projects and 6% of funding. Non-commercial projects are most commonly undertaken by universities or colleges, research institutions, industry associations and other non-profits, while commercial projects are, with few exceptions, undertaken by companies or partnerships. The leverage rate is lower for commercial projects than for non-commercial projects, which tend to have greater access to other sources of government funding.

In comparison to BDP projects, AIF projects tend to be much larger in value (average ACOA contribution of $2.3 million compared to $250,000 for BDP projects) and implemented over a longer time (four to five years versus 15 months). In part because of the size of ACOA’s contribution and lower degree of leverage, AIF projects were more likely than BDP projects to be cancelled (as opposed to proceeding as planned or being delayed, extended or reduced in scope) in the absence of ACOA funding.

AIF projects focus primarily on R&D activities (in the client survey, R&D activities accounted for 79% of AIF commercial project expenditures and 88% of AIF non-commercial expenditures). Expenditures associated with BDP projects (both commercial and non- commercial) are more evenly distributed between R&D and commercialization activities. As indicated below, clients receiving AIF funding were much more likely to indicate the primary focus of their project was the development of new technologies, products, services or processes than were BDP clients, whose projects were more likely to focus on later stages in the innovation process (e.g. demonstration, commercialization, marketing, and technology refinement, evaluation and adaptation).

Figure 2: Primary Focus of the Projects Surveyed, by Program Area

Source: Client survey. Recognizing the nature of some projects, the clients surveyed were allowed to select more than one primary area of focus. As a result, the numbers add up to more than 100%.

(D)

AIF commercial projects focus on the development of new products, processes and technologies or the refinement of existing ones across a range of sectors such as health, information and communications technology, renewable energy/energy efficiency, and manufacturing. According to QAccess data, expenditures of AIF commercial projects consisted primarily of wages (36%) and machinery and equipment (23%).

AIF non-commercial projects involve broader, systemic or cluster development initiatives (most commonly focused on health, agriculture, aquaculture and food, oil and gas, oceans, and pulp and paper) undertaken by universities or other non-profit organizations. They focus most often on the further development of applied R&D capacity, the implementation of R&D projects to develop new products, processes and technologies that can be commercialized by Atlantic Canadian companies, and the facilitation of technology transfer and commercialization. Leading examples include funding for Springboard Atlantic, which facilitates technology transfer from universities and colleges and promotes commercialization by companies in Atlantic Canada, work by Genome Canada in the early stage development of cod aquaculture, development of a research facility and implementation of a brain research program at Dalhousie, and a human genetics project at Memorial University. According to QAccess data, project expenditures consisted primarily of wages (41%), building structures (14%) and machinery and equipment (10%).

BDP commercial projects assist businesses in a wide variety of ways, such as developing new technologies, products and processes (in-house or through an R&D centre), refining existing technologies, products and processes, technology acquisition, skills development, and marketing. The projects focus on assisting businesses to grow and improve their competitive position (e.g. by achieving cost-savings through the acquisition of advanced technology). According to QAccess data, expenditures consisted primarily of machinery and equipment (47%), building and major renovations (12%), and wages (12%).

BDP non-commercial projects focus primarily on increasing the capacity and skills of participant organizations to undertake applied R&D and to commercialize new technologies, products, services and processes. Similar to AIF non-commercial projects, these projects support increased research capacity (e.g. by supporting infrastructure development), capacity for commercialization and the development of clusters in areas such as agriculture and food, aquaculture, health and medical technology, ocean technology, and bioscience). However, there is greater flexibility under the BDP to support a range of targeted activities (e.g. strategic planning, marketing studies, hiring of experts, staging of events, training and counselling). Leading examples include funding to establish a food product lab at a college, new research facilities at a research institute, a high-performance computing infrastructure for a network involving four universities in the region, a technology demonstration project, and a program that supports research internships involving students and recent graduates. Project expenditures consisted primarily of machinery and equipment (38%), building and major renovations (17%), and wages (16%).

There is some overlap between the AIF and the BDP in terms of clients. During the evaluation period, 46 of the 762 unique clients undertook both AIF and BDP projects under the IC sub-program, of which 26 were businesses [xxvi]. The case studies illustrate that the two programs are often complementary.

Research and infrastructure developed in earlier AIF projects was utilized or built upon in subsequent projects. In some cases, new technologies or products developed with AIF non- commercial and commercial assistance were later commercialized with BDP assistance. Some commercial clients reported that their project benefited from assistance provided by other projects supported through the IC sub-program (e.g. services provided by Springboard Atlantic or Learnsphere). In cases where a client undertook several projects, the projects often targeted similar outcomes (e.g. improved efficiency and business expansion).

4.2 Achievement of expected outcomes

This section presents the findings regarding impacts generated by projects funded under the IC sub-program in the immediate, intermediate and long term. The findings draw primarily from the client survey, the case studies and data extracted from QAccess.

The reported results are understated in that the impacts will continue to grow over time. Many of the projects had not yet generated impacts because they were still ongoing at the time of the evaluation. (Even though the research targeted older projects and overlapped with the previous evaluation’s scope, only 70% of the 277 projects surveyed were completed.) While some projects are able to report impacts before the project is fully completed, many require time post-completion before impacts such as increased revenues will occur. Furthermore, among projects that have already generated impacts, the effects tend to increase over time. For example, most clients that have already commercialized the results of their projects indicated that associated revenues are increasing over time (e.g. resulting revenues were higher last year than the previous fiscal year; a regression analysis indicates that revenue is positively correlated with length of time from project completion, although elapsed time is only one of the factors that affect resulting revenues).

The impacts of AIF projects are particularly understated because, as previously noted, they are more likely to focus on earlier stage R&D than are BDP projects (and therefore tend to require more time post-completion to generate impacts) and are much longer in duration (an average of four to five years in length versus 15 months for BDP projects). The survey sample was drawn from projects funded from 2005-2006 to 2012-2013 (the average start date was 2009), which means that many of the AIF projects were still under way (only 57% were completed at the time of the survey) or only recently completed.

The impacts of non-commercial projects also tend to be understated somewhat because it is easier to measure and attribute resulting impacts for commercial than non-commercial projects. Attributable impacts of non-commercial projects are often indirect (e.g. accruing to participants in project activities or spinoff companies rather than the client organization). For example, in the absence of a dedicated evaluation of its activities, it is not possible to report the aggregate impacts of projects such as Springboard Atlantic that work with a large number of institutions and/or companies.

The impacts have been extrapolated to the total population of projects funded under the IC sub-program during the eight-year period (2005- 2006 to 2012-2013) by aggregating the reported results, expressing the results on a per-dollar-of- ACOA-funding basis, and weighting the results by the level of ACOA funding provided by program and type of client. For ease of interpretation, the extrapolated results have been reported on a typical year basis by multiplying the impacts per dollar by the average level of funding approved by ACOA annually in each program area. [xxvii] Given the sample size, the estimates are reliable, although, as noted earlier, the reported results are understated in that the impacts will continue to grow over time.

4.2.1 Immediate outcomes

The intended outcomes of the IC sub-program in the short term include increased applied R&D activity; the attraction, development and retention of R&D staff; increased capacity for applied R&D; increased adaptation of technology; the establishment of networks and meaningful partnerships/alliances; and increased capacity (i.e. knowledge, skills, and infrastructure) for commercialization. The table below presents the percentage of surveyed clients that reported each type of impact and uses the data reported to forecast what the aggregate immediate impacts would have been, on an average annual basis, if all the clients had been surveyed about all of the projects undertaken during the period covered by the evaluation.

Table 5: Immediate Impacts of IC Sub-program Projects, Expressed on a Typical Year Basis

Impact	Percentage of Clients Reporting Impact to Date	Projected Impact of Projects Approved in a Typical Year
Investment in R&D	76%	$131 million
Investment in commercialization	67%	$57 million
Development of new partnerships	73%	547
New private-sector partnerships	71%	471
Technologies adapted	69%	373
Average full-time staff on the project	95%	1,035
Average part-time staff	61%	398
Newly hired staff	79%	728 (51% of all full-time and part-time project employees)
New staff involved in R&D activities	64%	368 (51% of all newly hired employees)
New staff who continued to be employed following project completion	77%	528 (71% of newly hired employees)

Note: Average annual ACOA assistance (2005-2006 – 2012-2013) was $82.4 million.

Clients were also asked to rate the progress made by their projects against the intended immediate outcomes. Average ratings vary by outcome and program area, depending on the extent to which particular projects targeted the intended outcome as well as the progress made to date. For example, most projects allowed staff who worked on the project to further develop skills and abilities while fewer targeted improving the organization’s marketing or business management practices. [xxviii]

Significant progress has been made toward the achievement of immediate outcomes, particularly in terms of increasing applied R&D activity and capacity; establishing networks, partnerships and alliances; and attracting, developing and retaining R&D staff. The results are discussed below.

On an annual basis, the IC sub- program has stimulated investment of approximately $131 million in applied R&D, including funding provided by ACOA and funding leveraged from other sources.

As a group, AIF clients reported that 85% of their project expenditures related to R&D, while BDP clients reported that 39% of expenditures related to R&D. The projects invested $1.59 in R&D activity ($1.24 and $2.19, respectively, for AIF commercial and non-commercial projects, and $1.32 and $1.06, respectively, for BDP commercial and non- commercial projects) for every dollar provided by ACOA.

Based on the average annual funding of $82.4 million approved by ACOA, the projects supported by ACOA annually are projected to invest approximately $131 million in applied R&D activity. Of this total, $51 million is invested by commercial clients. The level of investment in business R&D is significant; according to Statistics Canada data, the value of R&D funded by businesses in Atlantic Canada was $287 million in 2011. [xxix]

According to QAccess data, of the 1,131 projects approved by ACOA over the eight-year period of this evaluation, 44% focused specifically on applied R&D activity (i.e. the project was coded to technology development or new product and process development). These projects accounted for 67% of the total costs of the projects approved under the IC sub-program.

Projects approved annually under the IC sub-program have employed over 1,400 people (of whom 51% were newly hired) and facilitated skills development for about 350 students each year.

The development of staff skills and abilities (clients reported an average rating of 4.3 where 5 is “to a great extent”), further development of the skills and abilities of any students who worked on the project (average rating of 4.3), and the increased human resources development capacity of the organization (3.9) were the immediate impacts rated the highest by clients, particularly by non-commercial clients.

Ninety-five per cent of clients reported employing project workers on a full-time basis and 61% reported employing project workers on a part-time basis. The organizations (including those who did not employ project workers) employed an average of 7.3 people full-time and 2.9 people part-time to work on the project, of which 51% were newly hired and 27% were both newly hired and involved in R&D activities. Of the newly hired workers, 71% continued to be employed by the organization (or are expected to be) following completion of the project.

Thirty-five per cent of clients reported that their project involved students. Expressed on an annual basis, these projects have enabled an average of 350 students to develop their skills and abilities (mostly through projects undertaken by non-commercial clients, particularly those funded through the AIF).

In addition to staff and student development, capacity for applied R&D in Atlantic Canada has increased as a result of undertaking most projects in-house and further investing about $38 million in research-related equipment and infrastructure annually.

When asked to rate the impact of the project in increasing the capacity for applied R&D, on a scale of 1 to 5, where 5 is a major impact, clients provided an average rating of 3.5. Commercial and non-commercial AIF clients, which focus more directly on applied R&D capacity building than do BDP clients, provided higher average ratings of 4.0 and 4.3 respectively.

Of the research undertaken by the projects surveyed, 87% was undertaken in-house (including 74% undertaken entirely in-house) while 13% was undertaken entirely through an external organization (e.g. an R&D centre). According to QAccess data, $38 million of the average annual funding approved by ACOA was budgeted for capital investments in equipment and other infrastructure by projects that focused on applied R&D activity, of which 62% was invested in commercial projects and 38% in non-commercial projects.

The IC projects have facilitated the adaptation of approximately 370 technologies annually.

Sixty-nine per cent of clients surveyed reported that their project resulted in the adaptation of one or more technologies, although this was not necessarily the primary focus of the project. According to QAccess data, projects that focused specifically on technology adaptation, acquisition or diffusion accounted for 25% of projects, 13% of ACOA funding approved, and 17% of total project costs over the eight years. Clients provided an average rating of 3.8 with respect to the impact of the project on the adaptation of existing technology for use by their organization or others.

The IC sub-program has contributed to the establishment and expansion of networks, partnerships and alliances, including the development of an average of 547 partnerships and alliances annually, of which about 86% have been between the client and private-sector organizations.

A majority of clients (60%) reported that projects were undertaken in collaboration with other organizations. On average, clients reported an average of 3.3 other participants, 59% of which were private-sector businesses. Projects can result in new partnerships and alliances with project collaborators as well as with others, including spinoff companies, technology recipients, joint venture partners, investors, major buyers and others. For example, in one case study, a biotechnology start-up received assistance under the BDP to hire a consultant to assist in preparing a business plan that was used to attract major industry partners from the United States and Europe. In another case study, a manufacturing business partnered with an Atlantic university, with assistance under the AIF, to develop biodegradable packaging films and antimicrobial materials. The project led to discoveries that could have applications in other products and markets (e.g. surgical or medical equipment).

On average, clients reported that their projects led to the development of 3.7 new partnerships and alliances, 86% of which were with private-sector organizations. Partnerships and alliances among private-sector firms and institutions (e.g. universities, colleges and research institutions) are of particular relevance for AIF projects. According to a review of AIF project data, the proportion of projects involving partnerships between commercial organizations and institutions has increased over time for both projects led by institutions and those led by commercial clients, as shown in Figure 3.

Figure 3: Partnerships Between Institutions and Private- Sector Organizations in AIF Projects, by AIF Round [xxx]

Source: ACOA QAccess project management system, data extracted September 2013.

(D)

The IC sub-program has expanded commercialization capacity in Atlantic Canada, although more needs to be done. On an annual basis, it has stimulated investment of approximately $57 million in commercialization activities (including ACOA funding and that leveraged from other sources).

A majority of projects (67% of those surveyed) reported at least some expenditures related to commercialization. BDP clients reported that 46% of their project expenditures related to commercialization, while AIF clients reported that 14% of expenditures related to commercialization. The projects invested $0.69 in commercialization activities ($0.22 and $0.25, respectively, for AIF commercial and non-commercial projects, and $1.52 and $1.06, respectively, for BDP commercial and non- commercial projects) for every dollar provided by ACOA. Based on the average annual ACOA assistance of $82.4 million, IC projects are projected to invest approximately $57 million annually in commercialization.

Clients provided an average rating of 3.7 with respect to the impact of the project in strengthening the capacity of their organization to commercialize new technologies, products, processes or services (e.g. knowledge, skills and infrastructure related to commercialization). They provided a rating of 3.2 with respect to improving marketing efforts, and 3.2 with respect to improving business management practices. Commercial clients improved their commercialization capacity in a variety of ways, such as further developing internal staff capabilities (for example, as part of one case study project, a company retained the services of an executive search firm to recruit a qualified candidate for the new position of chief executive officer to drive the company’s expansion forward; the project resulted in increased employees, sales and equity financing for the company), engaging experts to develop business plans or attract investment, upgrading or expanding production capabilities, and developing marketing plans.

In addition to the support provided directly to commercial clients to facilitate capacity development, AIF and BDP funding has also been provided to non-commercial clients that deliver services related to commercialization. Non-commercial projects helped by facilitating access to services in areas such as technology transfer, angel investments and advisory services. For example, Springboard Atlantic received AIF funding to establish a network of university and community college technology transfer and industry liaison offices in Atlantic Canada involving 17 institutions as well as BDP funding to deliver innovation and commercialization workshops for start-up technology companies.

Though the capacity for commercialization has been further developed, most key informants indicated that further work is required. Both the literature and key informants noted that significant progress has been made over the past 10 years in developing R&D capacity and capabilities. The emphasis is now shifting somewhat from research to development. While ongoing investments will be required to sustain and further develop research capacity, the major challenge is to better facilitate the commercialization of new technologies, products, services and processes.

4.2.2 Intermediate outcomes

The intermediate intended outcomes of the IC sub-program include the development of new or improved technologies, products, processes and services, other benefits arising from the increased capacity for applied R&D (e.g. patents, technology disclosures and further investment in R&D) and commercialization of research results. The table below shows the percentage of clients reporting each type of outcome and uses the data to project the aggregate intermediate outcome on an average annual basis. As indicated, clients were much more likely to report progress in the development and commercialization of new technologies, products, processes and services and further investment in R&D than patents, technology disclosures, licensing agreements or spinoff companies.

Table 6: Intermediate Outcomes of IC Sub-program Projects, Expressed on a Typical Year Basis

Outcome	Percentage of Clients Reporting the Impact	Projected Impact of Projects Approved in a Typical Year
		$82.4 million
Development of new technologies, products, processes and services	67%	405
Issuance of patents	18%	114
Issuance of technology disclosures	16%	212
Further investment in R&D	58%	$66 million
Commercialization of technologies, products, processes and services	65%	521
Licensing or transfer agreements	27%	204
Development of spinoff companies	13%	41

Note: Average annual ACOA assistance (2005-2006 to 2012-2013) is $82.4 million.

Clients were also asked to rate the progress made by their projects against the intended intermediate outcomes. The average ratings were higher for the development of new or improved technologies, products, processes or services (4.2), commercialization (3.4) and further investment in R&D (2.8) than for licensing arrangements or transfer agreements (1.9), new patents filed (1.7), technology disclosures filed (1.6), or the establishment of spinoff companies (1.5), all of which are relevant to only a subset of the clients surveyed. [xxxi]

On an annual basis, the IC sub- program has supported the development of 405 new technologies, products, processes or services.

In the client survey, 68% of clients indicated that the project led to the development of new technologies, products, processes or services (an average of 3.3 per project), most of which were characterized as commercialized (46%) or market ready (32%). The 52 projects included in the case studies reported the development of 58 new products, processes and services as well as five new discoveries. Examples include biodegradable and antimicrobial packaging, an air ventilation system that controls both heat and humidity, a web-based artist management software, DNA tests that can be used to predict the reproductive longevity of livestock, and a vaccine platform that can have multiple applications.

On an annual basis, to date, the projects have resulted in 114 patents, 212 technology disclosures and $66 million in further investment in R&D.

Of the clients surveyed, 59% reported that the project led to further investment in R&D; 18% reported the issuance of one or more patents; and 16% indicated that it led to technology disclosures. The majority of patents (55%) and technology disclosures (66%) are associated with AIF non-commercial projects, while commercial projects (both BDP and AIF) accounted for the majority of investment in R&D (86%). Case study projects reported that 50 new patents were issued across a range of sectors and in multiple countries. It should be noted that patents and technology disclosures indicate that intellectual property has been created; however, they are not necessarily indicative of commercial potential. Further investment in R&D can be associated with further developing the technology that was the focus of the project or to undertake R&D on other related or non-related technologies.

Sixty-five per cent of clients reported progress toward the commercialization of technologies, products, processes or services, of which 68% reported increased revenues in the latest fiscal year.

On an annual basis, the projects have supported the commercialization of 521 technologies, products, processes or services to date. The projects have contributed to commercialization by supporting technology development and by supporting the commercialization of technologies that had already been developed. The majority of the impact (67%) is associated with BDP commercial projects.

Thirteen per cent of clients reported that the results of ACOA funded projects contributed to the establishment of spinoff companies.

On an annual basis, the projects have contributed to the establishment of 41 spinoff companies. The majority of the spinoffs resulted from non-commercial projects, including BDP and AIF projects.

On average, however, clients estimated that there was a 27% likelihood that the companies would have been established even if the project that received funding from ACOA had not been undertaken.

4.2.3 Long-term outcomes

The long-term outcome focuses on the growth and competitiveness of Atlantic SMEs, which can be reflected in indicators such as productivity improvements, revenues, employment and wages. As indicated in the table below, 80% of clients reported outcomes in terms of ongoing employment and wages, slightly over half reported increased revenues from commercialization, 42% reported cost savings or productivity improvements, and 30% reported the attraction of new investment.

Table 7: Long-Term Outcomes of IC Sub-program Projects, Expressed on a Typical Year Basis

Outcome	Percentage of Clients Reporting the Impact	Projected Impact of Projects Approved in a Typical Year
Cost savings/productivity improvements	42%	$26 million
Attraction of new investment	30%	$110 million
Revenues from commercialization	53%	$48 million
Ongoing employment	81%	750
Ongoing wages	81%	$37 million
Revenues of spinoff companies	9%	$26 million
Employment of spinoff companies	11%	208

Note: Average annual ACOA assistance (2005-2006 to 2012-2013) was $82.4 million.

When asked to rate the progress made by their projects against the selected outcomes, the long- term outcome rated highest (3.3) by clients was improvements in competitive position (e.g. market share, revenues or profitability of a business). Lower average ratings were provided with respect to cost savings/productivity improvements (2.5) and further investment in the organization by others (2.0), which is relevant only to some projects. [xxxii]

On a per-year-of-funding basis, clients report that the projects have generated $48 million in direct revenues to date and employment for 750 people. In addition, the spinoff companies have generated a further $26 million in revenues and employment for 208 people.

The clients that were surveyed reported that, in their latest fiscal year, they generated an average of $330,000 in revenues [xxxiii] from the commercialization of new technologies, products, processes or services resulting from the project. The average is up sharply from the prior year (when the clients reported an average of $160,000 in increased revenues), suggesting the revenues generated by the projects are still increasing. Of the aggregate revenues reported in the most recent fiscal year, 54% were generated from exports, 29% were generated from sales within Atlantic Canada, and 17% were generated from sales to other parts of Canada. On a per-year basis, the increased employment reported by the clients is equal to 750 positions, including employment with clients that are already generating revenues as well as those still working toward commercialization. Ninety per cent of the reported increase in revenues and 77% of the increase in employment was associated with BDP commercial projects.

Results from the case study cross analysis also point to the achievement of growth of Atlantic SMEs. In the case studies, clients reported the creation of 142 new jobs, the maintenance of 42 jobs, the expansion of 16 companies and increased revenues of $34 million.

On a per-year-of-funding basis, clients report that the projects generated approximately $26 million in cost savings or productivity improvements and the attraction of $110 million in new investment.

Of the cost saving and productivity improvements, 88% are associated with BDP projects, including 82% that are the result of BDP commercial projects. The savings most commonly resulted from new capital investments or the acquisition of new technologies or processes. According to QAccess data, 64% of the total budget for BDP commercial projects involved capital investments, particularly the purchase of new machinery and equipment and the development of new buildings and structures.

The $110 million in new investment was most commonly associated with the BDP program (57%). Some of the case study clients were particularly successful in attracting new investment. For example, one case study company that received AIF and BDP support was successful in attracting almost $40 million in equity and government assistance. Another company that also received support under the AIF and BDP recently secured a significant multi-million dollar investment from a single investor.

Over three quarters of clients reported that the projects enhanced their organization’s competitive position and nearly half reported broader social benefits.

Of those clients surveyed, 76% reported improvements in their organization’s competitiveness (a rating of 3 or more). These clients indicated that their competitive position has improved through strengthening company capabilities and capacity (23%), developing new markets or expanding existing markets (21%), improving productivity (17%), increasing market share (15%), introducing new products, processes or services (12%), and attracting new investment into the company.

Forty-four per cent of the clients reported the project generated social benefits such as environmental, safety or health benefits. Examples that were highlighted in the case studies include development and use of a vaccine to treat cancer patients in Halifax, development of renewable energy technology and reduction of reliance on plastics (electronic prescription cards).

Over the long term, ACOA’s funding of projects has supported the establishment and further development of a variety of industrial clusters.

While considerable work remains to be done, the results of the evaluation indicate that progress has been made since the 2010 evaluation of innovation programming, which found that clusters in Atlantic Canada were still in the early stages of development. The case studies illustrate how the BDP and AIF have contributed to progress in developing Atlantic clusters in the aerospace and defence, life sciences and biotechnology, ocean technologies, information and communications technology, and energy (particularly renewable energy) sectors. For example, since 2005 ACOA has provided support for seven BDP projects and one AIF project undertaken by PEI BioAlliance that have assisted the cluster in growing from 16 companies employing fewer than 450 employees to 38 companies employing over 1,100 people and generating $124 million in revenues; the cluster has developed an international reputation, which makes it easier to attract highly skilled labour to the region.

In addition, 11 BDP non-commercial clients surveyed reported contributing to cluster development through activities such as attracting partners, increasing international awareness and recognition of a cluster, promoting activities in rural areas, solving a technological or production issue that broadly affects the cluster, expanding facilities that can be used by businesses, and better enabling businesses to expand production, develop value chains and increase trade with other parts of Canada.

The economic analysis, using the Conference Board of Canada’s econometric model, estimates that the IC sub-program generated an increase in gross domestic product (GDP) of $5.49 per dollar of funding provided by ACOA over the 2007-2008 to 2012-2013 period.

The Conference Board of Canada’s (CBoC) econometric models were used to evaluate the economic impacts of IC projects funded under the AIF and the BDP. The analysis incorporates project expenditure data reported in QAccess as well as the results of the client survey regarding direct and spinoff revenues generated from the commercialization of products resulting from IC projects. The data has been adjusted for incrementality (i.e. the likelihood that the project would have proceeded with the same scope, at the same time and in the same location without government assistance) and attribution (i.e. the benefits attributed to ACOA are equal to the ratio of ACOA investment to total government funding).

The analysis focuses on the expenditures of IC projects from 2007-2008 to 2012-2013. During this period, IC project expenditures totalled $1.2 billion, of which $513 million was funded by ACOA. According to the CBoC model, these projects have contributed $2.8 billion to the GDP of Atlantic Canada, which represents an increase in GDP of $5.49 for every project dollar expended by ACOA. The projects have generated 32,000 person years of employment, which represents about one person year of employment for every $16,000 in ACOA funding provided. The impact of the projects will continue to grow as more projects reach the commercialization stage and as revenues and employment from commercialization increase over time.

The reported economic impact of the BDP projects to date is greater than the impact of the AIF projects. AIF projects tend to require more time post-completion to generate impacts because of their longer-term focus, their longer duration (an average of four to five years in length versus 15 months for BDP projects), and were less likely to be completed at the time of the survey (only 57% were complete). According to the CBoC model, BDP projects have generated $1.9 billion in Atlantic Canada GDP and 22,000 person years of employment to date, which represents an increase in GDP of $7.94 for every project dollar expended by ACOA and one person-year of employment for every $10,700 in ACOA funding provided. To date, AIF projects have generated $948 million in Atlantic Canada GDP and 12,000 person years of employment, which represents an increase in GDP of $3.41 for every project dollar expended by ACOA and one person-year of employment for every $27,800 in ACOA funding provided.

4.3 Contributing and constraining factors and lessons learned

The progress made to date toward the expected outcomes has been influenced by the characteristics of the projects supported, the incidence of highly successful projects, economic conditions, the capabilities of companies in the region, the nature of the programming and the use by clients of other resources supported by ACOA.

Major factors that influenced progress toward expected outcomes included the following:

The project type, focus, timing and sector. The evaluation involved a survey of projects about four years, on average, after they had been funded. As a result, projects positioned later in the innovation spectrum (e.g. focused on technology commercialization or adaptation) tended to generate a greater return to date per dollar invested in terms of cost savings, revenues and employment but have less of an impact in terms of incremental ongoing R&D expenditures. The sector in which the project takes place also affects the nature, timing and scale of impacts (for example, projects in the manufacturing and the information and communications technology sectors tended to report greater impacts to date while projects in the health & biotechnology sectors tend to involve a much longer path to commercialization due to the costs, the complexity of the research and regulatory processes. [xxxiv]
The incidence and magnitude of highly successful projects. Research, development and commercialization is an inherently risky activity. Government funding aims to reduce or spread this risk and encourage business investment in these activities. Given the risk profile, some projects will fail to meet their objectives (e.g. the R&D may not result in a marketable product, the proponent may be late to market, or the product launch may not be successful) while others may achieve only moderate success. The aggregate impact of any innovation program is largely determined by the presence of a comparatively small percentage of highly successful projects. These successes often require a range of assistance (e.g. R&D, development of production capabilities and international marketing) to realize their full potential. The reported impacts were reviewed to determine how many projects it took to account for at least half of the aggregate impact reported by clients. Three projects accounted for 52% of the reported spinoff companies, four projects accounted for 52% of the investment attracted, 10 projects accounted for 52% of the revenues from commercialization, 13 projects accounted for 51% of further investment in R&D, and 17 projects accounted for 51% of ongoing employment.
Economic conditions and the competitive environment (e.g. economic outlook, global competition, pace of change, market demand and access to other funding). The economic slowdown, which peaked in 2008, impacted the progress made by reducing the demand for funding, the willingness of businesses to take on the risk associated with new projects, access to funding for development and commercialization, and market demand for products. At the project level, clients emphasized that projects were successful when there was strong market demand, the product reached market ahead of the competition, and the product had clear competitive advantages. Availability of resources and investment also contributed to progress. For example, several key informants pointed to the growth of the oil and gas sector in Newfoundland and Labrador as contributing to increased provincial government fiscal capacity and availability of private-sector support for R&D. Some key informants, clients and case studies also indicated that the collaboration between federal and provincial partners contributed to the success of projects by providing additional resources and support to projects.
The client organization’s own business practices, knowledge and capabilities (e.g. management skills, reputation, contacts and expertise). A qualified team, management buy-in and drive, good reputation with existing contacts, established governance and management structure, expertise in the field, and knowledge of the sector and market needs, all contributed to the development of high quality new products, technologies, processes or services and success in attracting private-sector partners and investment and in generating revenues. According to some key informants, shortages in management skills, difficulties in attracting skilled workers (given the small population base), a shortage of experts that can facilitate commercialization (e.g. intellectual property lawyers, accountants and experts in business development), a risk-averse business culture and limited capacity for R&D continue to constrain innovation in the region.
The nature of ACOA’s IC programming. A few clients, case studies and key informants highlighted that ACOA’s ability to be flexible in making amendments, the low-risk, non-dilutive [xxxv] nature of ACOA’s financial support (which better enables clients to leverage investments from other federal programs, provincial programs and private-sector organizations), the wide range of financial and capacity building support provided by ACOA, and the capabilities and support of ACOA staff, all contribute to project success.
The interrelationship with other investments made by ACOA. In the survey, clients were asked first whether they used various resources supported by ACOA and, if so, about the impact of those resources on the success of their project (on a scale of 1 to 5, where 5 is to a great extent). Apart from other projects undertaken by the client organization, the resources that were used most commonly by the clients included trade shows and missions, the technology transfer services of Springboard Atlantic and incubators. The highest overall average rated services were incubators or technology accelerators (3.8), other resources or programs that received support from ACOA (3.8), other ACOA-supported projects undertaken by the client or others, and ACOA-supported trade shows or missions (3.5). [xxxvi]
Challenges associated with specific projects (e.g. limited budget, implementation delays and difficulties with partnering). Some clients identified insufficient resources as a major constraint to project success, noting that it is very difficult to predict budget needs for R&D projects. Case study participants and key informants noted that, prior to the recent changes, the approval (i.e. all phases of application processing) and contracting processes for AIF was too slow (in one case it took a project two years from application to project start) and the window to apply was too narrow and infrequent, which can make companies more reluctant to apply and can limit the commercial potential of projects. Numerous clients and case study participants identified implementation issues, including internal problems such as issues with management, staff turnover or exceeding budgets) and technical difficulties (i.e. related to research), as impeding the progress of projects. In addition, a few clients and key informants identified coordination issues with respect to partnering with other organizations, such as delays, differences in timelines across partners, and limited interaction between ACOA and some partners or between ACOA regional offices.

Key lessons learned include the importance of being responsive to business needs in program delivery, supporting capacity development, offering a range of assistance to clients that can be tailored to their specific needs, and leveraging resources.

Key informants and case studies identified a number of lessons learned.

Being responsive to business needs. Half of ACOA key informants noted that it is important to be responsive to business needs in delivering programming (e.g. continuous intake, more timely approval and contracting processes, allow for smaller funding amounts and shorter time frames under the AIF, and flexible program delivery). Case study participants also found that it is important to be responsive to business needs, including: 1) being flexible and responsive to changes in partnering with private-sector organizations given that it is difficult to predict or control changes in market circumstances; 2) moving swiftly and effectively when entering a new market space (several case studies involved the development of novel products/technologies that were expected to occupy a niche market space; however, that space was filled before the product made it to market); and 3) being responsive to the needs of new and emerging sectors. New and emerging clusters may have different needs from more traditional industries and their timelines for commercial success may vary.
Supporting commercialization capacity development. Some key informants and case study proponents found that ACOA has a role to play in enhancing the commercialization capacity and business skills of clients (i.e. in providing support for innovation ecosystems such as accelerators, incubators, mentorship programs, industry associations and venture capital forums as well as by assisting companies to expand internationally by facilitating participation in trade missions and connecting companies to trade officers).
Utilizing program tools to leverage private investment. A few key informants noted that tools such as repayable BDP contributions can be useful in leveraging venture capital investment. For example, the ACOA’s regional office in Nova Scotia encourages potential clients to seek venture capital investment based on ACOA’s provision of repayable BDP contributions. These types of arrangements have the potential to result in quality projects (since start-ups receive mentorship support from investors) and increase the likelihood of repayment.
Recognizing the time, resources and process of innovation. A few key informants noted that the path from innovation to commercialization is complex and can require significant resources and time. Sometimes ACOA assistance will support the first phase or platform in the development of a particular technology. Often clients require further assistance or follow-on investment to adapt technologies to new markets or private-sector partners.
Aligning with policy trends (e.g. federal, provincial and regional). Key informants also noted that there is a need to align with policy trends in innovation and commercialization. Federal innovation policy (e.g. the Jenkins Report) is placing greater emphasis on business, which aligns well with ACOA’s revised focus on commercialization of innovation.

4.4 Adequacy of performance measurement

ACOA staff have access to project monitoring information as well as some performance measurement data, which is compiled for the purpose of the Departmental Performance Report and/or reported in executive and operational dashboards.

Of the ACOA representatives who were interviewed, most believed that they had access to timely, reliable and relevant project monitoring and performance measurement data. Project level data is available for both the AIF (e.g. quarterly updates, annual reports and monitoring) and BDP (e.g. yearly assessments and monitoring), and ACOA has the ability to roll up high-level information. Using tools such as QAccess and dashboards that summarize QAccess data, information can be accessed on funded projects (e.g. commercial versus non-commercial, sectors, phases of R&D, repayments, partner dollars leveraged, etc.). In addition, representatives indicated that more detailed information can be accessed on a case-by-case basis such as the recent internal review of IC programming. Data is used primarily for monitoring and information gathering (e.g. service delivery levels, delinquencies and trends), managing the portfolio, adjusting the programming and Agency reporting.

Major challenges associated with the existing information system include difficulties in rolling up outcome level data, the time required before impacts are fully realized, difficulties in measuring certain outcomes, and some coding issues. ACOA’s Results Measurement Tracking System, which will allow account managers to track outcomes, holds promise for the availability of outcome data but it is not yet consistently used.

Gaps and challenges identified by key informants included:

An inability to roll up or standardize progress across innovation and commercialization projects as each project tracks different deliverables, outcomes and paths/timelines.
Time for impacts to be generated. Projects can take years to fully realize their impacts and project files may be closed prior to those impacts being realized. In addition, the policy focus and metrics may change before the impacts have been fully realized. (Programming used to focus more on R&D capacity building, and those impacts are just starting to be seen; however, the focus has already shifted to commercialization, which has different metrics.)
Difficulty measuring innovation and commercialization impacts. Innovation and commercialization projects often have spinoff impacts and other indirect or intangible impacts. It is difficult to capture and quantify the full impact of projects (e.g. a project may not result in commercialized technology that was planned but may help the company develop partnerships that result in the development of a different but related technology in the future).
The need to prioritize the entry of outcome data. Some representatives noted that the time and effort required to enter outcome information into the systems is often prioritized lower than other project monitoring activities. This results in some information not being entered or updated.

Other constraints noted include a lack of sufficient resources to roll up data and track progress (i.e. regions are busy delivering the programming) and certain key indicators (e.g. private capital leveraged) are not currently included in the Dashboard.

In addition, the case studies found some challenges and gaps with respect to performance measurement and information tracking. For most case studies, the project files contained an appropriate level of detail on project descriptions and targeted outcomes. However, several project files were missing information on progress toward outcomes and final reports. ACOA representatives interviewed noted that, in many cases, the final reports existed but were either not loaded into QAccess or were kept in paper format at regional offices. It was noted that the project filing system has been transitioning toward more consistent electronic filing in QAccess.

5.0 Findings: performance – efficiency and economy

This section examines the relationship between program costs and program outputs (operational efficiency) and outcomes (allocative efficiency) as well as how program costs are being minimized while optimizing the achievement of program outcomes (economy).

5.1 Efficient utilization of resources

Salaries and operating costs have averaged about 8.1% of the total IC sub-program budget, which is slightly higher than the previous evaluation period.

As indicated in Table 9, operations and maintenance (O&M) costs accounted for 8.1% of the program budget. O&M costs, as a percentage of total costs, has increased slightly since the last evaluation (from 7.8% for 2005-2006 to 2007-2008 to 8.3% from 2008-09 to 2012-13) due in part to a significant decline in grant and contribution (G&C) expenditures in 2011-12 and, to a lesser extent, an increase in O&M expenditures in 2009-10 and 2010-11 which has since been reversed.

Table 8: Program Delivery Costs by Fiscal Year

Year		2006-2007		2008-2009	2009-2010	2010-2011	2011-2012	2012-2013	Total
Program Expenditures
Operating Exp. ($M)	$1.6	$1.4	$1.7	$1.6	$1.6	$1.6	$1.0	$0.7	$11.3
Salary Exp. ($M)	$4.8	$5.4	$5.4	$5.4	$6.7	$6.0	$5.3	$6.0	$45.0
Total O&M Exp. ($M)	$6.5	$6.8	$7.1	$7.0	$8.3	$7.6	$6.3	$6.8	$56.2
G&C Exp. ($M)	$74.3	$82.9	$84.2	$88.3	$89.0	$96.2	$44.4	$81.5	$640.9
Total Expenditures ($M)	$80.7	$89.6	$91.4	$95.3	$97.3	$103.8	$50.7	$88.3	$697.1
Delivery Costs
O&M % of Total Exp.	8.0%	7.5%	7.8%	7.3%	8.5%	7.3%	12.4%	7.7%	8.1%
Total # Projects Approved	150	139	155	149	165	136	125	112	1,131
$ O&M per Project Approved	$43,110	$48,654	$45,980	$46,811	$50,203	$55,712	$50,270	$60,470	$49,733

Source: ACOA GX Financial information management system, data extracted May 2014.

5.2 Structures and mechanisms that support efficiency and economy

Various factors contribute to program efficiency including the governance structure, delivery approach, how resources are used, and use of performance information. Though some factors have constrained efficiency, recent actions have been taken to assess and minimize these barriers.

Most ACOA representatives interviewed perceive that the IC program is (or will be with the new changes resulting from the Innovation Review) designed and delivered efficiently and economically. It is important to note that this evaluation did not re-examine the design and delivery issues studied in the Innovation Review, nor can this evaluation speak to the efficiency and economy gains expected from these changes. Various factors contribute to the efficiency and economy include:

The regional governance structure. The ACOA governance structure, particularly the focus on regional program delivery, contributes to efficient design and delivery by enabling staff to be closely connected with local stakeholders and clients, and familiar with the local environment and needs. Regional offices tailor strategies to address regional priorities and work closely with provincial governments to align programming, policies, and strategies to develop a greater critical mass of support in targeted areas.
The delivery approach. Account managers assist clients in navigating the different ways in which ACOA can support their growth and meet their needs at different stages along the innovation continuum (e.g. from product development, to commercialization support, to assistance accessing international markets) and refer them to other services (e.g. incubators, mentorship programs, or other sources of provincial or federal financing).
Resource leveraging and repayability. The resources dedicated to innovation and commercialization programming are increased through leveraging (each ACOA dollar has been leveraged with $1.32 from other sources) and repayability. Many innovation projects are repayable or conditionally repayable (dependent upon the commercialization of products or achievement of outcomes). Of the IC project funding approved between 2005-2006 and 2012-2013, 80% had been disbursed as of September 2013 of which 31% was a repayable contribution and 8% was conditionally repayable. Of the amount disbursed, 32% of repayable contributions and 6% of conditionally repayable contributions had been repaid; in addition, the percentage will increase over time (e.g. 69% of repayable contributions and 20% of conditionally repayable contributions from 2005-07 have now been repaid).
The use of performance measures (e.g. dashboards). To help in program management, the dashboards contain useful information on key indicators such as delinquency rates, defaults, portfolio risk ratings, service standards, resource expenditures, and operational data. Plans are being implemented to expand dashboards to include additional data on operations, managing new business, and managing the G&C portfolio.

Key informants, clients, and case study representatives identified that several factors associated with the AIF program constrained program efficiency. These same challenges (e.g. timeliness of AIF approvals and frequency of funding rounds) were also identified during the Innovation Review, which led to the announcement of changes to the programming on July 16, 2014. [xxxvii] Some of the major changes include a move to continuous intake of AIF proposals with bi-monthly or quarterly decision points, increased emphasis on shorter term (i.e., less than three years) AIF projects, increased focus on technically sound projects with commercial potential, integrated branding of innovation programming and intake of project applications, increased promotion of technology adaptation projects, focus on value-added role of industry advisors, increased reliance on regional oversight, and the development of a commercialization element under the BDP.

The results reported in this 2014 evaluation of the IC sub-program are comparable and, in some respects, superior to those reported in the 2010 evaluation.

The 2010 and 2014 used similar methodologies (surveys of clients, case studies, and review of QAccess data), although the 2010 evaluation relied primarily on case studies for assessing economic impacts (e.g. revenues) rather than a mixture of case studies and surveys. In addition, in 2010, certain data (e.g. technology disclosures and spinoffs) were reported only for AIF projects. As indicated in the following table, the two evaluations reported broadly similar data with respect to leverage, revenues and commercialization with the AIF projects in the most recent evaluation reporting higher average figures with respect to patents, transfer agreements, technology disclosures, and spinoff companies.

Table 9: Selected Results from the 2010 and 2014 Evaluations

Impact	2010 Evaluation	2014 Evaluation
Impacts Per Dollar of ACOA Funding
Leverage ¹	$1.44	$1.32
Revenues ²	$0.46	$0.53
Impacts Per Organization Surveyed
Commercialization of technologies, products, processes and services ²	2.5	2.9
Issuance of patents ⁴	0.3	0.8
Licensing or transfer agreements ⁴	0.4	1.2
Issuance of technology disclosures ³	1.7	6.0
Development of spinoff companies ³	0.2	0.8

Sources from 2010 and 2014 Evaluation: ¹ QAccess Data; ² 2010: Case Studies of 36 projects with revenues allocated over five years; 2014: Client survey; ³ 2010: Survey of 88 organizations funded under the AIF; 2014: Client survey of AIF projects only; ⁴ 2010: Survey of 339 organizations funded under the BDP and AIF; 2014: Client survey

5.3 Delivery costs

The IC sub-program compares most closely with the National Research Council’s Industrial Research Assistance Program (IRAP) program, although there are some significant differences. The IC sub-program has comparatively lower delivery costs as a percent of G&Cs provided, although the costs per project approved are similar.

Amongst innovation and commercialization programs, the IC sub-program is most similar to NRC-IRAP, which also employs a regional delivery model in providing G&Cs to businesses to undertake R&D and innovation activities as well as funding for regional and national organizations that support development and commercialization of technologies by SMEs. NRC-IRAP differs from the IC sub-program in that it provides technical and business advisory services and its G&Cs are, on average, smaller and positioned closer to the commercialization end of the innovation spectrum (and, therefore, generate greater impacts in the short-term). These differences are reflected in the costs of the two programs. As indicated below, delivery costs as a percentage of total expenditures are much lower under the IC sub-program (8.3%) than NRC-IRAP (33.4%), although costs per project are similar.

Table 10: Comparison of Delivery Costs, ACOA’s IC Program and NRC-IRAP Program

Program	ACOA IC	NRC-IRAP
Program Expenditures (2007-2008 to 2011-2012)
Total O&M Exp. ($M)	$36.2	$222.7
Total G&C Exp. ($M)	$402.2	$444.5
Total Expenditures	$440.4	$667.2
Delivery Costs
O&M as a Per cent of Total Exp.	8.3%	33.4%
Total Projects Approved	730	5,197
O&M per Project Approved	$49,652	$42,843

5.4 Alternative modes of delivery

The most common suggestions to maintain or improve program efficiency and economy were to streamline processes, reduce timelines for approval, and pursue a more integrated approach to program delivery, which are in line with the changes currently being made to the programming.

Other common suggestions were to:

Ensure that the programming remains flexible and responsive to client needs. Clients, case study representatives and key informants stressed the importance of flexibility and responsiveness, with some suggesting changes such as allowing for more than one project to be supported at the same time, broadening the definition of eligible costs, and reducing reporting requirements for smaller projects.
Continue to support commercialization capacity building. Key informants, clients and case study representatives highlighted the importance of support mechanisms such as providing support to non-profit organizations and events that are part of the innovation eco-system, providing direct assistance to SMEs to allow them to access skilled expertise, and facilitating partnerships between early stage SMEs and key players in global networks.
Further strengthen communication and alignment with partners. Key informants, clients, and case study representatives noted that ACOA needs to maintain close coordination and communication with partners in order to reduce the potential for duplication, assist clients with path finding and leveraging of funding, coordinate support to strategic sectors, share information about the available programs, and play a convening role to promote cross-provincial coordination in pursuing pan-Atlantic regional objectives.
Develop better mechanisms to roll up and report on project outcomes. Several groups of ACOA representatives suggested that ACOA should develop better mechanisms to roll-up project outcomes, employ a broader range of data sources (e.g. project reports, surveys, and anecdotal evidence via case studies) on an ongoing or annual basis, revise the performance indicators (e.g. track both standard and unique indicators, report on key success factors, and monitor trends in macro-economic indicators, etc.), and improve its systems and tools (e.g. QAccess and dashboards).
Further support ACOA staff in program delivery (e.g. facilitate ongoing sharing of best practices in delivery approaches, training, and capacity building). Key informants noted that there are opportunities to further share and benefit from best practices developed in different ACOA regions. It was noted that processes that brought staff from different regions together (e.g. for training or to participate in AIF sector teams) helped staff to gain knowledge and make connections. Given the increasing complexity of projects, key informants and clients noted that ACOA staff should receive more training related to innovation and commercialization (e.g. training on IP agreements), and a few key informants noted that account managers should be provided with more guidance and training with respect to due diligence (at the contract and implementation stages), understanding transactions and reading financial statements.

6.0 Conclusions and recommendations

6.1 Conclusions

The major conclusions arising from the evaluation of the Innovation and Commercialization sub- program are as follows.

Relevance

There is a strong, continued need for the IC sub-program given the importance of innovation and commercialization to regional competitiveness, the need to build on the progress made in recent years, and the fit between the program and client needs.The IC sub-program responds to this need by committing significant funding for innovation and commercialization and by placing emphasis on the development of SMEs and clusters in key strategic sectors. Most clients indicated that the assistance provided by ACOA met the needs of their organization and that only 4% of projects would have proceeded in the absence of ACOA support.
Innovation and commercialization are major drivers of business productivity and economic competiveness. Although Atlantic Canada, and more specifically the IC sub-program, has made progress in strengthening commercialization capacity and increasing investment in R&D over the past decade, more needs to be done. The region continues to lag behind other regions of Canada on key innovation and commercialization indicators (e.g. level of investment in R&D, level of R&D performed by the private sector, access to venture capital and highly qualified personnel, proportion of federal research grant funding awarded, etc.). Factors that constrain innovation performance include the industrial structure, the small domestic market, significant interprovincial and regional disparities, and a risk averse business culture.
Changes have recently been made to better align the IC sub-program with client needs and evolving federal government priorities. These changes may impact demand for funding, types of clients and projects, results and efficiency.
The Agency has access to the information it needs to adjust its programming and strategies in response to changes in the innovation and commercialization landscape. ACOA has monitored the changing innovation and commercialization landscape in a recent review of innovation programming and adjusted its programming to better meet existing and emerging needs, as well as reflect evolving federal government priorities (e.g. changes to programming with a renewed focus on meeting the needs of businesses, such as more timely delivery, and a greater focus on funding commercialization).
The IC sub-program tends to complement rather than duplicate innovation and commercialization programming delivered by other organizations as well as other ACOA sub- programs due to its focus on commercialization, levels of assistance, ability to take on risk, regional delivery and comprehensive support. Given regular changes in the programming environment, ensuring that the IC sub-program is well coordinated requires ongoing effort.The IC sub-program also works closely with other ACOA programming in providing direct and indirect support to clients along a continuum of development. Just over half of IC clients received funding from more than one ACOA sub-program.
The IC sub-program is differentiated from other programs that promote innovation and commercialization in Atlantic Canada by its strong focus on commercialization, levels of financial assistance available, support for riskier projects and clients, and regional delivery. To minimize the potential for overlap and to leverage support for clients, ACOA coordinates activities with other programs. Seventy-five per cent of clients reported their projects received funding from sources other than ACOA. Although some areas of overlap were identified by key informants (e.g. some provincial governments are developing programming similar to ACOA’s), they were seen as minimal and normally addressed through communication and collaboration.
The IC sub-program is consistent with the strategic priorities, roles and responsibilities of the federal government and ACOA as well as the strategies and approaches implemented by federal organizations in other regions of Canada and in other jurisdictions.The IC sub-program aligns well with the roles and responsibilities of ACOA and the federal government in Canada as well as other jurisdictions. The sub-program is consistent with the ACOA Act, which established the Agency as the primary federal government department responsible for economic development efforts in Atlantic Canada. The focus, priorities and roles of the sub-program are also consistent with the approaches implemented by federal government organizations in other jurisdictions (e.g. placing a high priority on innovation and commercialization, shifting their emphasis toward market “pull” innovation policy, broadening their definition of innovation, developing clusters, placing greater priority on SMEs, promoting partnerships between the private sector and research organizations, addressing commercialization gaps, and coordinating programming across different levels of government and partners).
The objectives of the IC sub-program are consistent with federal government priorities, particularly as stated in the Science and Technology Strategy 2007, Budget Plan 2014, speeches from the Throne and the 2011 Jenkins Report. ACOA has taken steps to ensure its programming aligns with federal government priorities by promoting public-private partnerships, targeting priority sectors and increasing the focus on commercialization. The objectives of the sub-program are also consistent with ACOA’s focus on helping Atlantic SMEs to be more innovative, productive and competitive.

Performance – effectiveness

The IC sub-program has resulted in extensive direct and leveraged investments in innovation and commercialization projects which, in turn, have generated a wide range of impacts in the immediate and intermediate term. Over the long term, the sub-program has expanded innovation and commercialization capacity and contributed to the growth and competitiveness of SMEs in Atlantic Canada.
An average of over $80 million in project funding was approved annually by ACOA, and that amount was leveraged with $1.32 in funding from other sources for every dollar contributed by ACOA. On an annual basis, the IC sub-program has stimulated approximately $131 million in applied R&D expenditures, including investments of $38 million in research-related equipment and infrastructure, $57 million in commercialization expenditures, project employment for over 1,400 people, the development of 547 partnerships and 405 new technologies, products or processes, the adaptation of 370 technologies, and the commercialization of 521 technologies, products or processes that, as per the latest fiscal year, had generated $48 million in direct revenues and $26 million in spinoff revenues as well as 750 direct jobs and 208 spinoff jobs. These reported results are understated in that the impacts will continue to grow as more projects reach the commercialization stage and revenues and employment from commercialization increases over time. On an annual basis, the IC sub-program has also led to 114 patents, 212 technology disclosures, $66 million in further investment in R&D, $26 million in cost savings or productivity improvements, and the attraction of $110 million in new investment. Over the long term, ACOA funding of projects has supported the establishment and further development of various industrial clusters in Atlantic Canada, including aerospace and defence, life sciences and biotechnology, ocean technologies, information and communications technology, and energy (particularly renewable energy) sectors. The economic analysis, using the CBoC’s econometric model, estimates that the IC sub-program generated an increase in GDP of $5.49 for every dollar expended by ACOA over the 2007-2008 to 2012-2013 period.
Major factors that influence the progress made toward expected outcomes include: the characteristics of projects; the incidence and magnitude of highly successful projects; economic conditions and the competitive environment; the client’s business practices, knowledge and capabilities; the design and delivery of the sub-program; the interrelationship between investments made by ACOA; and project-specific challenges.The main factors that constrained progress include challenges associated with the project (e.g. lack of budget, implementation delays, difficulties with partnering), inefficiencies in program delivery (e.g. one entry date and delayed contracting times for AIF projects), and economic, social and geographic challenges (e.g. global competition and the pace of change, lack of available financing, lack of capacity for applied R&D, lack of skilled personnel, geographic distance to markets, and a risk averse business culture).
Progress is influenced by project type, focus (e.g. projects that are more focused on commercialization generate a greater return per dollar invested), timing (e.g. impacts increase over time), sector (e.g. biotechnology projects tend to have longer commercialization time frames than information and communications technology projects), the client’s own business practices (e.g. technical and business management skills), the design and delivery of ACOA programming (e.g. comprehensive, flexible, low-risk support that promotes leveraging and industry involvement), and the economic environment (e.g. market demand and the availability of resources).
Key lessons learned include the importance of being responsive to business needs in program delivery, supporting capacity development, offering a range of assistance to clients that can be tailored to their specific needs, and leveraging resources.
Major lessons learned during the implementation of projects and programming included the importance of being responsive to business needs (e.g. being flexible and responsive to change in partnering with private-sector organizations, moving swiftly and effectively when entering a new market, and being responsive to the needs of new and emerging sectors), supporting commercialization capacity building (e.g. supporting innovation and commercialization ecosystems such as accelerators, mentorship programs, industry association and venture capital forums), utilizing program tools to leverage private investment (e.g. the approach of the Nova Scotia regional office in using repayable BDP to leverage venture capital investment for projects), recognizing that innovation can be a lengthy and complex process, and aligning with partners and policy trends (e.g. federal, provincial and regional).
While ACOA staff have access to project monitoring and performance reporting information, there are some specific challenges that need to be addressed (e.g. inability to roll up outcome data and difficulties in coding impacts).The main gaps and challenges associated with performance measurement include a lack of ability to roll up or standardize progress or outcomes across the portfolio; time to see impacts; difficulty in measuring innovation and commercialization impacts (e.g. difficulties in coding projects to innovation versus other programs such as productivity and growth); and the relative priority afforded to the entry of outcome data in the QAccess system
Most ACOA representatives interviewed reported having access to timely, reliable and relevant project monitoring and performance measurement data in terms of project-level data (e.g. quarterly updates, annual reports and risk monitoring) and high-level information using the dashboards and QAccess (e.g. sectors, repayments, partner dollars leveraged, etc.). More detailed information can be accessed on a case- by-case basis and through more thorough reviews. Data is used primarily for monitoring and information gathering (e.g. service delivery levels, delinquencies, trends), managing the portfolio (e.g. ensuring balance in terms of level of risk), adjusting the programming, and Agency reporting.

Performance – economy and efficiency

The existing governance structure, delivery model and funding mechanisms appear to support the efficient and economical delivery of the programming (O&M costs averaged about 8% of total program expenditures). Further improvements are expected from recent changes to the programming. There may be opportunities to further improve efficiency and economy by sharing tools, processes, promising practices and lessons learned across regions.Salaries and operating costs, as a percentage of the total program budget for the sub-program total costs, have increased slightly since the last evaluation (from 7.8% for 2005-2006 to 2007- 2008, to 8.3% from 2008-2009 to 2012-2013). This is due, in large part, to a significant decline in G&C expenditures in 2011-12. There may be opportunities to further improve delivery across regions (e.g. through standardization and the sharing of tools, processes, promising practices and lessons learned).
Various factors contribute to the IC sub-program’s efficiency, including the regional governance structure (e.g. allows regions to tailor strategies and focus to needs), the client-centric delivery approach (e.g. offering a range of support mechanisms), the increased availability of resources through leveraging, the sharing of risks through repayability, and the collection and use of high-level performance information (e.g. dashboards) to monitor portfolio performance and make investment decisions.
Actions have recently been taken to address program barriers. While the changes are expected to benefit efficiency and economy, the results will need to be carefully monitored.
The main efficiency constraints identified relate to the timeliness of approvals (i.e. at all phases of application processing) and contracting process for the AIF (e.g. it can take too long for AIF projects to be approved and contracted, the window to apply for funding is too small, and the funding rounds are too infrequent). An internal review was recently undertaken of innovation programming and, as of July 2014, changes have been implemented that aim to improve program efficiency (e.g. moving to a continuous intake system for the AIF, using a generic Agency application form for both the AIF and BDP, lowering the AIF private-sector threshold, and introducing a new BDP commercialization element). Plans for measuring the impact of these changes are being implemented.

6.2 Recommendations

The major recommendations arising from the evaluation of the Innovation and Commercialization sub-program are as follows.

Carefully monitor the recent programming changes to assess the impact and determine the need for further changes or adjustments.Some changes have also been made to the delivery structure. One of the benefits of the regional structure and level of regional autonomy that exists within ACOA is the opportunity it provides to share experiences, knowledge and information gained with respect to effective approaches, client needs, recent developments, lessons learned and promising practices. This information, however, is often not formalized, with preference given to informal means of communication. The regional structure can create silos between offices. The AIF sector committees provided a valuable venue for the sharing of information between account managers and between the regions overall. However, with the restructuring of the AIF program, these sector committees are no longer in place. A key function of the new and existing governance structures should be to facilitate and strengthen internal communication and formalize the sharing of information and lessons learned across regions. Opportunities should also be explored for facilitating formal information sharing across the sub-programs within ED.
The impacts of the recent programming and structural changes should be carefully monitored to assess the impact on the demand for funding, the characteristics of clients and projects supported (e.g. the relative balance between investments in R&D and investments in commercialization capacity), approval times, risk, resulting impacts, and ongoing communication and sharing of information, both internally and externally, regarding client needs, recent developments, lessons learned and promising practices. Existing and new governance structures at the project level (e.g. commercialization champions, use of industry advisors and technical advisors in conducting due diligence, etc.) and at the strategic level (e.g. DG Operations Committee, ED Directors Committee, AIF Advisory Board, etc.) will assist in monitoring and assessing the risks and impacts of the changes.
ACOA has recently introduced significant changes to the IC sub-program. Based on the results of its recent review, the Agency renewed its commitment to IC and is strengthening its programming by launching a new BDP commercialization funding element as well as by moving to a continuous intake system and lowering the minimum funding threshold to better enable the AIF to meet the needs of business. With these changes, the programming is expected to better operate at the speed of business and to place a greater emphasis on commercialization (without excluding targeted investments in R&D capacity).
Revise the Performance Measurement Strategy (PMS) for the IC sub-program to reflect the recent program changes, and improve the utility of the system for both performance reporting and management planning by integrating outcome data into monitoring systems.To address these issues, ACOA should consider employing a mixture of standard and unique indicators. (A few core, high-level performance indicators can be applied to each IC project; these standard indicators can be augmented, as necessary, by additional project-specific indicators.) Consideration should also be given to moving toward a more client-focused approach to performance measurement with respect to project outcomes (rather than a strictly project-by-project approach), aligned with ACOA’s client-centred approach to ED program delivery. This approach would reflect the full range of assistance provided to a client, recognizing that the majority of IC clients receive ACOA support multiple times and under multiple sub-programs. ACOA should also explore opportunities to mine the existing data systems to monitor performance, assess trends and inform future programming decisions. Existing data could be augmented by collecting and rolling up information on client needs or the progress of clients in achieving milestones (e.g. from development to commercialization) as part of client tracking activities.
Program logic models are being revised and the results will need to be reflected in a revised PMS. Difficulties in meaningfully rolling up project outcomes have meant that the impact information reported by clients has not been used extensively for either performance reporting or management planning. A low priority is often placed on inputting impact or outcome information reported by clients into QAccess.

Appendix A: management action plan

Evaluation	Recommendation	Planned Actions	Responsibility	Target Date	Status
Innovation and Commercialization Management Action Plan (MAP) approval date: Pending	Recommendation 1: Carefully monitor the recent programming changes to assess the impact and determine the need for further changes or adjustments	The Agency will monitor programming changes and their impacts and will ensure ongoing communication by making use of the existing reporting mechanisms and governance structure, including the Pan Agency Review Committee (PARC), the director deneral of Operations (DG Ops), directors of Enterprise Development (ED), Atlantic Innovation Fund (AIF) regional coordinators and the Commercialization Champions Committee. More specifically: The Agency will perform program oversight and examine program performance by monitoring program direction, engagement activities with clients and external stakeholders, uptake, results and commitments, and client and project characteristics. Oversight by PARC will be key to ensuring ongoing program monitoring and dialogue among regions and head office on challenges and opportunities. At its annual face-to-face meeting, the AIF Advisory Board will review the impact of the AIF and provide advice on policy matters to the minister and president. The Agency will formalize the ongoing discussion with regional AIF coordinators and the AIF Secretariat by establishing monthly conference calls. The Agency will maintain its ongoing dialogue with its federal and provincial counterparts and with the other numerous stakeholders in the innovation space. Results of monitoring and oversight will be reviewed by the president and the executive committee on an annual basis. The Agency will complete a risk assessment exercise on the new AIF due diligence process.	Director general, ED in collaboration with the director general, Policy, the regional and head office director generals and ED directors. Director, Innovation and Entrepreneurship	Ongoing monitoring of the planned actions to ensure their effectiveness will occur over a two-year period ending December 31, 2016. March 31, 2015	Plans are in place for monitoring impacts resulting from the implementation of the recent programming changes and to ensure ongoing communications with internal and external stakeholders. The assessment exercise is in the planning stages
Recommendation 2: Revise the Performance Measurement Strategy (PMS) for the IC sub- program to reflect the recent program changes and improve the utility of the system for both performance reporting and management planning.	In Spring 2014, ACOA engaged Performance Management Network to assist in undertaking a review of the Performance Measurement Strategy for ED and its sub programs. Work will continue internally to refine and finalize the PMS, taking into consideration the findings of the evaluation as well as Agency plans regarding performance measurement for ED moving forward.	Director, Innovation and Entrepreneurship in collaboration with regional and head office ED directors.	October 31, 2015	The PMS review is currently under way.

Evaluation

Recommendation

Planned Actions

Responsibility

Target Date

Status

Innovation and Commercialization

Management Action Plan (MAP) approval date: Pending

Recommendation 1: Carefully monitor the recent programming changes to assess the impact and determine the need for further changes or adjustments

The Agency will monitor programming changes and their impacts and will ensure ongoing communication by making use of the existing reporting mechanisms and governance structure, including the Pan Agency Review Committee (PARC), the director deneral of Operations (DG Ops), directors of Enterprise Development (ED), Atlantic Innovation Fund (AIF) regional coordinators and the Commercialization Champions Committee. More specifically:

The Agency will perform program oversight and examine program performance by monitoring program direction, engagement activities with clients and external stakeholders, uptake, results and commitments, and client and project characteristics. Oversight by PARC will be key to ensuring ongoing program monitoring and dialogue among regions and head office on challenges and opportunities.
At its annual face-to-face meeting, the AIF Advisory Board will review the impact of the AIF and provide advice on policy matters to the minister and president.
The Agency will formalize the ongoing discussion with regional AIF coordinators and the AIF Secretariat by establishing monthly conference calls.

The Agency will maintain its ongoing dialogue with its federal and provincial counterparts and with the other numerous stakeholders in the innovation space.
Results of monitoring and oversight will be reviewed by the president and the executive committee on an annual basis.

The Agency will complete a risk assessment exercise on the new AIF due diligence process.

Director general, ED in collaboration with the director general, Policy, the regional and head office director generals and ED directors.

Director, Innovation and Entrepreneurship

Ongoing monitoring of the planned actions to ensure their effectiveness will occur over a two-year period ending December 31, 2016.

March 31, 2015

Plans are in place for monitoring impacts resulting from the implementation of the recent programming changes and to ensure ongoing communications with internal and external stakeholders.

The assessment exercise is in the planning stages

Recommendation 2: Revise the Performance Measurement Strategy (PMS) for the IC sub- program to reflect the recent program changes and improve the utility of the system for both performance reporting and management planning.

In Spring 2014, ACOA engaged Performance Management Network to assist in undertaking a review of the Performance Measurement Strategy for ED and its sub programs. Work will continue internally to refine and finalize the PMS, taking into consideration the findings of the evaluation as well as Agency plans regarding performance measurement for ED moving forward.

Director, Innovation and Entrepreneurship in collaboration with regional and head office ED directors.

October 31, 2015

The PMS review is currently under way.

Appendix B: overview of ACOA

Program	2012-2013 Actual Spending ($M)	Sub-program	Key Indicators of Performance
Enterprise Development	$179.9	Innovation and Commercialization Productivity and Growth International Business Development	Survival rates of ACOA-assisted firms Sales growth of ACOA- assisted firms
Community Development	$88.5	Community Mobilization Community-based Business Development Community Investment Infrastructure Programming	Percentage points by which the growth in sales of CBDC-assisted clients exceeds that of unassisted firms Percentage of projects that are successful in achieving the intended objectives
Policy, Advocacy and Coordination	$12.4	Policy Advocacy Coordination	Atlantic regional economic policies and programs that respond to regional development opportunities
Internal Services	$34.9	Governance and Management Support Resource Management Services Asset Management Services
Total	$315.7	Impact on Atlantic Canada's GDP for every $1 of ACOA expenditure in direct support of business Labour productivity growth of ACOA-assisted firms

Sources: Atlantic Canada Opportunities Agency 2012-13 Departmental Performance Report and 2013-14 Report on Plans and Priorities.

Appendix C: evaluation issues and questions

Evaluation Issues and Questions
Relevance
Issue 1: Continued Need for the Program
1.1	To what extent does the Innovation and Commercialization sub-program continue to address a demonstrable need?
1.2	To what extent is the sub-program aligned with the existing and emerging needs of its stakeholders?
1.3	What other mechanisms exist to address these needs? Where are the areas of integration, overlap, duplication and gaps?
Issue 2: Alignment with Government Priorities
2.1	To what extent is the sub-program aligned with federal government priorities and expectations? To what extent is it aligned with ACOA’s strategic outcome, priorities and strategy?
Issue 3: Alignment with Federal Roles and Responsibilities
3.1	To what extent is the sub-program aligned with federal roles and responsibilities?
Performance
Issue 4: Effectiveness
4.1	How and to what extent is the sub-program achieving expected outcomes (immediate, intermediate and long-term outcomes)? What has been the contribution of this sub-program to the innovation ecosystem of Atlantic Canada?
4.2	To what extent has ACOA’s gradual transition toward a greater focus on commercialization (e.g. in project selection) contributed to the achievement of expected results?
4.3	Incrementality: What impact would the absence of the sub-program have on projects/initiatives?
4.4	What are the facilitators and barriers to achieving expected outcomes (from ACOA and client perspective)? To what extent are these being addressed?
4.5	What lessons have been learned in the implementation of the components of the sub- program? How can these lessons contribute to future and/or other programming?
4.6	What unintended outcomes have been achieved?
4.7	To what extent are the sub-program’s performance measurement and reporting structures effective in reporting on the achievement of outcomes? How and to what extent is the performance information used by clients and by ACOA?
Issue 5: Efficiency and Economy
5.1	In the context of the results being achieved, how and to what extent are the resources allocated to the sub-program efficiently and economically utilized?
5.2	Are there more efficient and economical ways of achieving expected results, taking into consideration current and alternative approaches to delivery, sub-program governance, barriers to efficiency/economy, and the application of best practices and lessons learned?

Appendix D: innovation and commercialization sub-program logic model

Reach	CLIENTS / DIRECT TARGET Universities Firms Research institutions OTHER STAKEHOLDERS Industry groups / associations Federal science-based departments National programs (e.g. CFI, NSERC, TPC) Provincial governments CO-DELIVERERS Advisory Boards Science / Industry advisors	Activities and Outputs	Financing innovation / commercialization projects
		Immediate / Direct Outcomes	Increased applied R&D activity Increased capacity for applied R&D Increased adaptation of technology Establishment of networks and of meaningful partnerships/alliances Attraction, development and retention of R&D staff Increased capacity [e.g. knowledge, skills, infrastructure] for commercialization
		Intermediate / Indirect Outcomes	Development of new or improved technologies / products / processes / services [Benefits of] increased applied research capacity in Atlantic Canada in both the private sector and institutions Research results are commercialized with benefits to Atlantic Canada
		Long-term Outcome	Strengthen Atlantic Canada’s innovation and commercialization capacity
		Enterprise Development	Improved growth and competitiveness of Atlantic SMEs
		Strategic Outcome	A competitive Atlantic Canadian economy

Adapted from: ACOA, Performance Measurement. ACOA 2012-2013 Program Activities and Sub-activities Logic Models, February 2012.

Appendix E: other programs supporting innovation and commercialization in Atlantic Canada

Organization	Description	Types of Support	Target Groups	Resources and Funding Amounts
FEDERAL GOVERNMENT PROGRAMS
National Research Council of Canada Industrial Research Assistance Program (NRC-IRAP)	Federal funding program that assists SMEs to undertake innovation and applied R&D activities	Business development support, mentorship Technical advice and funding to SMEs – Industrial Technical Advisors (ITAs) Non-repayable contributions for early stage R&D	SMEs, start-ups	National G&C budget: $173.2 million in contributions for IRAP in 2012-2013 [xxxviii] Atlantic Canada G&C budget: about $24 million to $30 million or about 15% of total budget (budgets are allocated regionally) Average $40,000 per project (ranges from $5,000 to $1 million) Atlantic Canada staff: 50 FTEs, including 28-32 ITAs based throughout the Atlantic region and two concierges G&C for NRC increased by $104.1 million, from $143.6 million in 2011-2012 to $247.7 million in 2012-2013, as a result of an additional $110.0 million per year in funding to double the Industrial Research Assistance Program as announced in Canada's Economic Action Plan 2012. [xxxix] Also, new initiatives have recently been announced, including the Canada Accelerator and Incubator Program ($58.5 million over five years to a limited number of accelerators and incubators), the Business Innovation Access Program ($20 million program to connect SMEs to institutions to overcome commercialization barriers such as planning, marketing, etc.), and the Concierge Service, which provides advice and information about available programming
Natural Sciences and Engineering Research Council (NSERC)	Federal funding agency focused primarily on post-secondary institution-led discovery research in the natural sciences and engineering fields	Grants for natural sciences and engineering discovery and applied research projects Scholarships for research students	Mostly post-secondary institutions and researchers Discovery research and applied research	National G&C budget: $866.6 million in grants and scholarships in 2012-2013; [xl] 350 FTEs Atlantic Canada G&C awarded: $50 million-70 million on average Atlantic Canada office: four FTEs; $50,000 for travel, office supplies; $125,000 regional fund grant money for events Grants range from $10,000-25,000 to multi-million dollar Idea to Innovation Grant – contribution funding for post-secondary institutions/researchers to accelerate R&D of promising technology and to promote its transfer to a new or established company ($5,000-$350,000) [xli] Collaborative Research and Development Grants – contribution funding for post-secondary institutions and private-sector partners to undertake applied R&D [xlii]
Canadian Institutes of Health and Research (CIHR)	Federal funding agency for health research	Funds open/investigator-initiated research and top-down strategic research goals	Individuals (researchers, knowledge users), and institutions (health research and research training in universities, health-care institutions, voluntary health-sector organizations and research institutes)	National G&C budget: $852.8 million in grants for research projects in 2012-2013; [xliii] three to four FTEs dedicated to program delivery Atlantic Canada G&C awarded: Information not available Examples of programs: Proof of Principle Program (POP) – market/business development of academic research: $100,000-$250,000 per year for one to two years Industry-Partnered Collaborative Research Operating Grant (IPCR) - $250,000 per year for five years Science to Business – encourages PhDs to pursue an MBA Collaborative Health Research Grants – partnership with NSERC, driving innovative interdisciplinary collaborative research projects
Social Sciences and Humanities Research Council (SSHRC)	Federal funding agency that supports post-secondary-based research and training in the humanities and social sciences	Funding for research grants, scholarships, post doc research Some research focuses on public policy and programs that contribute to innovation and economic development In some cases, more directly involved with firms (business strategy and market development)	Researchers, students, post-secondary institutions, and community or non-profit organizations	National G&C budget: $337.0 million in 2012-2013 Atlantic Canada G&C awarded: $20.1 million in 2012-2013 Awards can range from $25,000 for scholarships to large research awards (e.g. $2.5 million over seven years) [xliv]
Networks of Centres of Excellence (NCE)	Federal funding agency jointly administered by Canada's three granting agencies: CIHR, NSERC and SSHRC in partnership with Industry Canada and Health Canada	Centres of Excellence for Commercialization and Research (CECR) program – provides funding for the establishment and operating costs of centres of excellence that match clusters of research expertise with businesses to undertake innovation and commercialization activities	New or existing not-for-profit corporations created by universities, colleges, not-for-profit research organizations, firms, and other interested non-government parties	20 FTEs total staff in Ottawa (0.5 FTEs administering grants) Centres of Excellence receive about $6 million to $15 million over 5-10 years Atlantic Canada has three funded centres that will receive $33.7 million in funding over four to five years and support about 10 FTEs: Leading Operational Observations and Knowledge for the North (LOOKNorth): $7.1 million for 2011-2016 Marine Environmental, Observation, Prediction and Response Network (MEOPAR): almost $25 million for 2012-2017 Children and Youth in Challenging Contexts (CYCC): $1.6 million for 2011- 2015 [xlv]
Canada Foundation for Innovation (CFI)	Federal funding agency that provides funding for infrastructure necessary to conduct leading-edge research	Funding for infrastructure necessary to conduct leading-edge research (e.g. state-of-the-art equipment, laboratories, databases, specimens, scientific collections, computer hardware and software, communications linkages and buildings)	Universities, colleges, research hospitals and non-profit research institutions	National G&C awarded: $422.8 million awarded in 2012 Atlantic Canada G&C awarded: $5.2 million in 2012 Awards range from about $5,000 to multi-millions (most about $150,000) [xlvi]
Business Development Bank of Canada (BDC)	Federal Crown Corporation that focuses on financial and other services for businesses	Provides loans to target sectors Venture Capital (VC) funds	Businesses Target sectors designated by government: ICT, Innovative health care, clean tech/scalable energy	National loans: $4.1 billion in new loans in 2013 [xlvii] National VC: $135 million to innovative health care, $100 million for clean tech/renewable energy for 2013 Atlantic Canada loans: $166 million in loans in 2013 Atlantic Canada offices: about 150 FTEs across N.L., N.S., and N.B.
Canada Revenue Agency (CRA)	Federal government department responsible for administering tax laws and incentive programs	Scientific Research and Experimental Development Tax Incentive Program (SR&ED) – gives claimants cash refunds and/or tax credits for their expenditures on eligible R&D work done in Canada	Canadian-controlled private corporations, other corporations, and individuals/trusts	A Canadian-controlled private corporation (CCPC) can earn a refundable Investment Tax Credit (ITC) of 35% on qualified SR&ED expenditures. This refundable ITC is 100% refundable on qualified SR&ED current expenditures and 40% refundable on qualified SR&ED capital expenditures incurred before 2014, up to a maximum threshold of $3 million of qualified SR&ED expenditures for SR&ED carried out in Canada. A CCPC can also earn a 15% non- refundable ITC on any amount over that threshold [xlviii]
Industry Canada	Federal government department responsible for helping to make Canadian industry more productive and competitive in the global economy	Strategic Aerospace and Defence Initiative (SADI) – repayable contributions to R&D projects in the aerospace, space, defence and security sectors Technology Demonstration Program (TDP) – non-repayable contributions in support of large-scale technology demonstration projects in the aerospace, defence, space and security sectors Mitacs – coordinates collaborative industry-university research projects focused on skills development	SADI: firms of all sizes in the aerospace, space, defence and security sectors TDP: original equipment manufacturer (OEM) or a Tier 1 company Mitacs: graduate students in R&D fields, businesses	SADI: Project sizes range from $276,000 to $300 million TDP: one to three new projects are expected to be approved per year; the maximum funding that may be allocated in any one year across projects is $54 million ;[xlix] Mitacs: The 2014 Budget proposes $8 million over two years to Mitacs to expand its support for industrial research and training of post-doctoral fellows. Federal funding for Mitacs programs will also be made available to eligible not-for-profit organizations with an economic orientation. “Mitacs will become the single delivery agent of federal support for post-doctoral industrial R&D fellowships, as the NSERC Industrial R&D Fellowships program will be wound down, with its resources redeployed to other priorities within the Council, including basic discovery research.” [l]
Department of Finance	Federal government department responsible for economic, fiscal, tax, social, security, international and financial sector policies and programs.	Venture Capital Action Plan – funding to establish new VC funds, and recapitalize and invest in existing VC funds in Canada	Private sector investors and provincial governments High growth businesses	The Government will pursue a comprehensive action plan for deploying the $400 million in new capital over the next seven to ten years. The Government's approach recognizes the need to demonstrate that Canada's innovative firms represent superior return opportunities, and that private-sector investment and decision making is central to long-term success. With this in mind, the Venture Capital Action Plan will make available: $250 million to establish new, large private-sector-led national funds of funds (a fund of funds portfolio consists of investments in several venture capital funds) in partnership with institutional and corporate strategic investors, as well as interested provinces Up to $100 million to recapitalize existing large private-sector-led funds of funds, in partnership with willing provinces An aggregate investment of up to $50 million in three to five existing high-performing venture capital funds in Canada [li]
Public Works and Government Services Canada	Federal government department that acts as the government’s central purchasing agent, linguistic authority, real property manager, treasurer, accountant, integrity adviser, and pay and pension administrator	National Shipbuilding Procurement Strategy Defence Procurement Strategy (DPS)	Businesses	The $35 billion Strategy was first announced in June 2010. Requests for proposals were released in February 2011, and the first agreements were signed in 2012 New Defence Procurement Strategy launched in February 2014. The Department of National Defence published its first annual Defence Acquisition Guide in June 2014 The Jenkins Panel found that procurement programs accounted for only 1% of R&D expenditure in 2010-2011. Since then, the government has prioritized procurement as a fast-track commercialization strategy In Atlantic Canada, ACOA is implementing several initiatives under the Atlantic Shipbuilding Action Plan [lii]
Sustainable Development Technology Canada (SDTC)	Not-for-profit foundation that finances and supports the development and demonstration of clean technologies; reports to Parliament through the Minister of Natural Resources Canada	Finances and supports the development and demonstration of technologies that improve the quality of the climate, air, water or soil.	Mostly businesses (80% of consortia are industry-led)	SDTC operates two funds: the SD Tech Fund™ for projects relating to air, water and soil, and The NextGen Biofuels Fund™ for first-of-kind large demonstration facilities for renewable fuels. It reports to Parliament through the Minister of Natural Resources [liii]
PROVINCIAL GOVERNMENT PROGRAMS
Innovacorp	Nova Scotia Innovation Corporation – Provincial Crown Corporation under the Nova Scotia Department of Economic Development	Early stage VC/equity Business incubation and mentoring Start-up competitions/awards Programming and education regarding entrepreneurship (IP protection, sales, legal advice, etc.)	Early stage high growth companies, start-ups	35 FTEs working in four offices in Halifax (N.S.) and Sydney (C.B.) Invested $5.18 million in 2012-2013 in 12 technology companies [liv] $250,000-$2 million per client in equity investments $40,000-$100,000 typical prize amount for competition winners
New Brunswick Innovation Foundation (NBIF)	New Brunswick independent, not-for-profit corporation with initial capitalization from the provincial government that invests in new growth-oriented companies and applied research activities.	Grant contributions to research institutions Equity investment to start-up companies Research talent development [lv]	Start-ups/SMEs Research institutions	Grants: $2 million per year ($10,000 to $500,000 per project) Equity investments to start-up companies: $2 million per year ($100,000 to $500,000 per project on average) $1 million for research talent development ($10,000 per project)
Nova Scotia Department of Economic and Rural Development and Tourism	Nova Scotia provincial government department responsible for promoting economic growth	Various innovation and productivity funding programs that contribute to costs such as acquiring advanced machinery or equipment skills certifications; a voucher program, which SMEs can use to acquire help from universities/colleges to make their business more innovative; and student employment wage rebate programs [lvi]	SMEs	The Capital Investment Incentive contributes 20%, up to a maximum of $1 million, toward the cost of technologically advanced machinery, clean technology, equipment, software and hardware The Workplace Innovation and Productivity Skills Incentive provides funding to companies to encourage investment in skills development and certification ($5,000 to $10,000 and above if employer shares costs) Productivity and Innovation Voucher Program assists SMEs to acquire help from universities/colleges to make their business more innovative (Tier 1 vouchers are worth up to $15,000 for new applicants, and Tier 2 vouchers are worth up to $25,000 to build on work undertaken with the use of Tier 1 vouchers). In 2013-2014, 51 vouchers were awarded, including 41 Tier 1 and 10 Tier 2 vouchers (maximum of $865,000 in funding)
Innovation PEI	PEI economic development agency operating at arm’s-length from the provincial government	Programs focused on human resources (labour rebate for new incremental growth) Marketing, IT establishment, productivity improvement Administers an annual pilot and discovery award and a development and commercialization award [lvii]	SMEs ICT, bio science, aerospace advanced manufacturing, and alternative energies sectors	35 FTEs In 2012-2013, $16.1 million was approved to support 729 projects with total budgets of $87.2 million [lviii] Smaller grants range from $10,000-$40,000 Pilot and Discovery award: $25,000 Development and Commercialization award: $100,000 Innovation and Development Labour Rebate: 25% of eligible salaries
Research and Development Corporation (N.L.)	Newfoundland and Labrador agency focused on promoting applied R&D, operating at arm’s-length from the provincial government	Several grant and contribution programs aimed at both business and universities.	Businesses and universities	$3.8 million operating for 2013 $20 million in grants for 2013 ($17.6 million for academic stream, $3.2 million for businesses) [lix] Grants range from $15K to $5 million Business-led Programs: R&D Vouchers – provides businesses access to local, national and international scientific and technical equipment, expertise and research facilities ($15K max contribution) R&D Proof of Concept – increases the technical capacity of businesses to perform R&D and reduces the financial risk of R&D activities ($250,000 max contribution) Industrial R&D Fellowships – enables businesses to hire post-doctoral researchers ($30,000 annually) Academic-led Programs: Collaborative R&D – strengthens institutional R&D capacity through supporting business-academic collaboration ($800,000 max contribution) Ignite R&D – strengthens institutional R&D capacity through funding for new researchers ($100,000 max contribution) Leverage R&D – strengthens institutional R&D capacity through leveraging against other funding sources ($500,000 max contribution) Strategic Sector Programs: Petroleum R&D Accelerator, GeoEXPLORE (2011-2013), and ArcticTECH (2012-2014) [lx]
New Brunswick Department of Economic Development	New Brunswick department responsible for driving the provincial government’s economic development strategy through key priorities	Liaises between ministry and the N.B. Research and Innovation Council (responsible for delivering programming) Trying to develop Springboard-like model to provide unified and collaborative commercialization education across post-secondary education ecosystem Offers repayable and non-repayable funding to businesses for innovation and productivity improvements (e.g. capital, equipment and salary costs)	Businesses Strategic sectors	Six FTEs in Research and Development Innovation Unit $80 million part of innovation agenda over five years; different levels of programs, research technology assistance, innovation challenges – encourage universities and private-sector partners, many of them co-funded with ACOA INNOV8 Program – companies developing intellectual property, specialized software, hardware, equipment, or performing R&D or prototyping (up to $25,000 per project and $100,000 per company per year) Financial Assistance to Industry Program – funding for capital expenditures and working capital to enable the establishment, expansion or maintenance of eligible industries; assistance may be provided in the form of a loan guarantee or direct loan NB Growth – assistance for businesses in targeted sectors; funding for capital costs associated with the establishment or expansion/diversification of those eligible businesses or related to improving the productivity or competitiveness of those businesses and salary costs; non-repayable contributions of up to $100,000 for establishment, and up to $60,000 for expansion [lxi]
Regional Development Corporation (N.B.)	New Brunswick provincial Crown Corporation that plans, coordinates and implements regional and economic development initiatives for the provincial government	Mostly grant based for applied research, loans for projects undertaken by businesses aimed at commercialization	Businesses and post-secondary institutions	Manages the Northern New Brunswick Economic Development and Innovation Fund: $200 million for April 2011-March 2015. Projects range from $25,000-$500,000 Manages the Miramichi Regional Economic Development and Innovation Fund: $50 million from April 2011-March 2015. Projects range from $25,000-$500,000 [lxii]
RPC	New Brunswick provincial Crown Corporation research organization	Provides research service and business support on a fee-for-service model Research targeted to business needs, technical services (measurement, analysis, inspections) [lxiii]	Private businesses	95 FTEs (from technicians to PhD level) $10 million per year in sales
OTHER NOT-FOR-PROFIT / REGIONAL PROGRAMS
Springboard (Atlantic Canada)	Regional commercialization and industry liaison network for Atlantic Canada based in Nova Scotia	Helps to maintain a commercialization network of 18 institutions by supporting Industry Liaison and Tech Transfer Offices Provides engagement opportunities/develops best practices for linking industry and university for commercialization outcomes; support programs/advisory services regarding commercial development, patent filing, export markets; and some direct funding of businesses [lxiv]	Member institutions Businesses	Five central staff Contributions for the salaries of 30 Industry Liaison or Tech Transfer Officers at post-secondary institutions in Atlantic Canada Industrial Undergraduate Student Research Awards Program (up to $6,500 per student) Innovation Mobilization Program (Industry Engagement, Patent and Legal, Proof of Concept, and Marketing Support) (from $15,000 to $50,000 per project) $14 million budget for three years Funded through ACOA (AIF) as a special initiatives project
BioNova (N.S.)	Industry association for life science sector in Nova Scotia	Advocates on behalf of sector, provide leadership Educates stakeholders about the presence and success of the sector Full life cycle of programs/services for member companies [lxv]	Businesses in the life science sector in Nova Scotia	$260,000 in 2013-2014 for business support services/programs Four FTEs
BioNB (N.B.)	Bioscience industry association for New Brunswick	Provides coaching, mentoring, business development support, sector experience, technological translation, access to grants, networking [lxvi]	Entrepreneurs and researchers in the bio sector (20 core members, 30 non-core)	$300,000 annual operating budget Three to five FTEs Operates off an endowment fund (not government or membership funded)
The Next Phase (N.S.)	Entrepreneurship training/mentorship program based in Nova Scotia	A series of workshops and one-on-one coaching that prepares entrepreneurs to develop their business concept, attract investment, and pitch to potential angel investors [lxvii]	Entrepreneurs Angel investors	$240,000 annual operating costs
Propel ICT/ Launch 36 (N.B.)	Intensive accelerator program designed to quickly move selected entrepreneurs to an investor-ready position; based in N.B.	Runs two accelerator program sessions per year with the goal of successfully launching 36 companies over a three-year period. [lxviii]	High growth start-ups	Information not available
Volta – Halifax Startup House (N.S.)	Nova Scotia-based not-for-profit that provides mentorship and support to new start-ups and potential entrepreneurs.	Mentors and supports start-ups to establish and grow Develops community-driven education and support Provides space and networking for start-ups [lxix]	Start-ups and new entrepreneurs	Information not available
Venn Innovation Inc. (formerly Tech South East) (N.B.)	Intermediary not-for-profit membership-based organization advancing the tech sector in N.B.	Advisory/networking services for tech companies Program connects youth with firms to foster new tech entrepreneurs [lxx]	Tech companies in N.B. Potential entrepreneurs, students	Information not available
Wallace McCain Institute (N.B.)	Member-based entrepreneurial support network in N.B. based at the University of New Brunswick	Educational, peer-group formatted Conferences, speakers, networking [lxxi]	Entrepreneurs across Atlantic Canada	$1.2 million operating budget Four FTEs Some direct support from ACOA to run programming Fee-for-service programs operate at break-even 175 members paying between $100 and $20K per year
Pond-Deshpande Centre for Innovation & Entrepreneurship (N.B.)	Mentorship centre for entrepreneurs in N.B., based at University of New Brunswick	Funding support/grants Conferences, events Mentorship Accelerator program/development dialogue for social enterprise Student ambassador program [lxxii]	Entrepreneurs Social enterprises Students/youth	$200,000 in grants annually ($500-$1,000 each for Student Catalyst Fund; $5,000 -$15,000 in Social Innovation Fund and Technology Ignition Fund) 4.5 FTEs
Genesis Centre (N.L.)	Support network to help high-growth, knowledge-based start-ups in N.L. become “investor ready,” based at Memorial University of Newfoundland	Mentorship, technical resources at MUN, physical resources (office equipment, etc.) [lxxiii]	High-growth, knowledge-based start-ups in N.L.	Information not available

Appendix F: client survey ratings tables

Table 11: Importance of Factors When Considering Applying to ACOA versus Another Source

Question: On a scale of 1 to 5, where 1 is not at all important and 5 is very important, how important were the following considerations in your decision to apply for funding from ACOA rather than from another organization?

Question	BDP Non-commercial	BDP Commercial	AIF Non-commercial	AIF Commercial	Average
Level of funding available	4.5	4.6	4.9	4.9	4.7
Eligibility requirements	4.3	4.3	4.2	4.3	4.3
Timeliness of project approvals	4.1	4.3	4.1	4.0	4.3
Definition of eligible expenditures	4.4	4.3	4.2	3.8	4.2
Unsuccessfully approached other funding sources	3.7	4.1	3.9	3.9	4.1
Other funding sources were not available	4.2	4.0	4.1	3.7	4.0
Familiarity with ACOA	4.4	4.0	3.6	3.9	4.0
The application process	3.2	3.5	3.6	3.4	3.5
Reporting requirements	3.2	3.2	3.5	3.1	3.2
Non-financial assistance provided by ACOA	3.3	3.3	3.1	2.9	3.2

Table 12: Ratings of Immediate Impacts Generated by Undertaking the Project

Question: To what extent did undertaking the ACOA-assisted project result in: (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent)

Question	BDP Non-commercial	BDP Commercial	AIF Non-commercial	AIF Commercial	Total
Improving business management practices used by your organization	3.2	3.3	3.3	2.9	3.2
Improving the marketing efforts of your organization	3.1	3.3	3.1	2.7	3.2
Strengthening the research capacity of your organization (e.g. human resources, facilities and equipment)	3.9	3.3	4.3	4.0	3.5

The development of new partnerships and alliances between your organization and others	4.3	3.5	4.3	3.5	3.7
The expansion of your existing networks, partnerships and alliances	4.2	3.5	4.2	3.6	3.7
Strengthening the capacity of your organization to commercialize new technologies, products, processes or services (e.g. knowledge, skills and infrastructure related to commercialization)	3.7	3.8	3.5	3.7	3.7
The adaptation of existing technology for use by your organization or others	4.2	3.7	4.4	3.1	3.8
Increasing the human resource development capacity of your organization	4.5	3.8	4.4	4.0	3.9
The development of the skills and abilities of any students who worked on the project	4.8	4.1	4.7	4.5	4.3
The development of the skills and abilities of your staff	4.6	4.3	4.6	4.4	4.3

Table 13: Ratings of Intermediate Impacts Generated by Undertaking the Project

Question: To what extent did undertaking the ACOA-assisted project result in: (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent)

Question	BDP Non-commercial	BDP Commercial	AIF Non-commercial	AIF Commercial	Total
The establishment of one or more spinoff companies	1.8	1.4	2.0	1.2	1.5
Filing technology disclosures	1.9	1.3	2.8	2.3	1.6
Filing new patents	1.8	1.6	2.3	1.9	1.7
Licensing arrangements or transfer agreements signed between your organization and others	1.8	1.8	2.6	2.1	1.9
Further or ongoing investment by your organization in R&D	2.9	2.7	2.6	3.4	2.8
The commercialization of new technologies, products, processes or services by your organization or others	3.0	3.5	3.1	3.4	3.4
The development of new or improved technologies, products, processes or services	4.1	4.1	4.5	4.5	4.2

Table 14: Ratings of Long-term Impacts Generated by Undertaking the Project

Question: To what extent did undertaking the ACOA-assisted project result in: (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent)

Question	BDP Non-commercial	BDP Commercial	AIF Non-commercial	AIF Commercial	Total
Further investment in your organization by others	2.8	1.8	2.6	2.0	2.0
Social benefits such as environmental, safety or health benefits	2.2	2.3	2.7	2.5	2.3
Cost savings and/or improvements in productivity for your organization	1.9	2.8	1.7	2.0	2.5
Improvements in your organization’s competitive position (e.g. market share, revenues or profitability)	2.8	3.5	2.5	3.1	3.3

Table 15: Use and Ratings of Other Resources Supported by ACOA

Question: Did your organization make use of any of the following resources supported by ACOA when implementing the project or acting upon the results? If so, to what extent did the resource contribute to the success of your project and the resulting impacts? (on a scale of 1 to 5, where 1 is not at all, 3 is to some extent, and 5 is to a great extent)

	BDP Comm. (n=202)		BDP Non-comm. (n=28)		AIF Comm. (n=24)		AIF Non-comm. (n=23)		Total (n=277)
	% used service	Rating	% used service	Rating	% used service	Rating	% used service	Rating	% used service	Rating
Incubators or technology accelerators	9%	4.0	25%	3.4	8%	3.0	13%	4.3	11%	3.8
Other resources or programs that received support from ACOA	20%	3.7	32%	4.1	17%	3.0	17%	4.0	21%	3.8
Other ACOA-supported projects undertaken by your organization or others	25%	3.6	54%	4.1	21%	3.4	22%	4.0	27%	3.7
ACOA-supported trade shows or missions	20%	3.6	32%	3.3	42%	3.1	26%	3.2	23%	3.5
Learnsphere (The Commercialization Consulting and Mentoring Program)	6%	3.4	4%	3.0	21%	3.4	13%	3.7	8%	3.4
Other services by Springboard Atlantic	6%	2.6	18%	4	4%	2.0	30%	3.9	9%	3.2
Technology transfer office/ industry liaison offices through Springboard Atlantic	16%	1.9	21%	4.2	8%	2.0	39%	4.0	18%	2.6

Appendix G: profile of IC projects, by program and type of client, 2005-2006 to 2012-2013

	AIF		BDP
	Commercial	Non-Commercial	Commercial	Non-Commercial
Projects	90	89	790	162
Projects	8%	8%	70%	14%
Unique Clients	24	52	610	76
Leading Types of Clients	Incorporated company (98% of projects)	University (70%) Other non-profit (12%)	Incorporated company or partnership (99%)	University (49%) Other non-profit (24%) Industry assoc’n (17%)
ACOA Funding	$209.5 million	$204.9 million	$198.0 million	$40.1 million
ACOA Funding	32%	31%	30%	6%
Average ACOA Assistance	$2.33 million	$2.30 million	$250,686	$247,766
Total Costs	$437.9 million	$463.0 million	$499.1 million	$112.8 million
Avg. Project Budget	$4.87 million	$5.20 million	$631,766	$696,030
Leverage Rate	$1.09	$1.26	$1.52	$1.81
Incrementality (per client survey) in the absence of funding:	Project would have Proceeded as planned (0%) Been cancelled (35%)	Project would have: Proceeded as planned (0%) Been cancelled (70%)	Project would have: Proceeded as planned (6%) Been cancelled (27%)	Project would have: Proceeded as planned (0%) Been cancelled (64%)
Sources of Funding	ACOA (48%) Other federal (3%) Provincial (4%) Client/other (46%)	ACOA (44%) Other federal (11%) Provincial (5%) Client/other (39%)	ACOA (39%) Other federal (3%) Provincial (10%) Client/other (47%)	ACOA (36%) Other federal (8%) Provincial (12%) Client/other (44%)
Average Length	4.1 years	5.0 years	1.3 years	1.2 years
Sub-Elements Funded (% of ACOA funding)	In-house development of product and process (86%) Product and process development – R&D centre (11%) Technology acquisition (2%) Study (1%).	Technology development – applied research (84%) Commercialization (14%) Technology development – basic research (1%) Technology diffusion (1%)	In-house product/process development (37%) Technology acquisition (36%) Development product/process – R&D centre (5%) Hiring expertise (3%) Other (19%)	Infrastructure and equipment (53%) Commercialization (15%) Applied technology dev’t (9%) Promotion and awareness (5%) Other (18%)
Expenditures by Category (as % of total project costs)	Salaries/wages (36%) Machinery/equip. (23%) Building (5%) Other capital (4%) Other operating (31%)	Salaries/wages (41%) Building/renov. (14%) Machinery/equip. (10%) Other capital (1%) Other operating (34%)	Machinery/equip. (47%) Building/renov. (12%) Salaries/wages (12%) Other capital (5%) Other operating (23%)	Machinery/equip. (38%) Building/renov. (17%) Salaries/wages (16%) Other capital costs (7%) Other operating (22%)
Percent of Project Budget Spent On (per client survey):
R&D	79%	88%	39%	35%
Commercialization	14%	10%	45%	35%
Locale of R&D Activities (per client survey):
In-house	77%	71%	74%	74%
External	5%	5%	14%	22%
Both	18%	24%	13%	4%

Sources: QAccess data and client survey

Footnotes

[i] Atlantic Canada Opportunities Agency, 2012-2013 Departmental Performance Report.

[ii] Government of Canada, Budget 2014. http://www.budget.canada.ca/2014/docs/plan/ch3-2-eng.html.

[iii] Innovation and Commercialization full- time equivalent data was not available for 2005-2006 and 2006-2007.

[iv] The economic and community development activities formerly undertaken by Enterprise Cape Breton Corporation have transitioned over to ACOA. However, given the timing and scope of the present evaluation, Cape Breton is reported as a separate region.

[v] The delivery process described applies to the period being evaluated. Between fall 2012 and spring 2014, ACOA’s Policy and Programs branch conducted a formal, ten-year retrospective analysis of the Agency’s IC activities and coordinated consultations with stakeholders across Atlantic Canada. Based on the results, the Agency renewed its commitment to IC and is strengthening its programming by launching a new BDP commercialization funding element as well as by streamlining the BDP and AIF application intake (one application form), changing the AIF approval process to a continuous intake system and lowering the minimum funding threshold to better enable the AIF to meet the needs of business. These changes are expected to impact the demand for funding, the types of clients and projects supported (i.e. increased emphasis on commercialization), the resulting impacts and efficiency.

[vi] OECD/Eurostat, Oslo Manual: Guidelines for Collecting and Interpreting Innovation Data, 2005.

[vii] World Economic Forum, The Global Competitiveness Report 2013–2014.

[viii] Government of Canada, Innovation Canada: A Call to Action. Review of Federal Support to Research and Development – Expert Panel Report, October 17, 2011.

[ix] RE$EARCH Infosource Inc., 2011

[x] Statistics Canada, Table 358-0161 – Business enterprise research and development (BERD) characteristics, by industry group based on the North American Industry Classification System (NAICS), provinces and territories, annual.

[xi] Some conventional indicators such as patenting cover only a small portion of the innovation spectrum. In recognition of this, discussions by the federal government are shifting to a new paradigm of innovation. This point is reviewed further in a discussion of alignment with federal roles and responsibilities.

[xii] According to a review of online grant award databases, the level of funding awarded to Atlantic institutions has increased from about $30 million in 2001-2002 to $50 million in 2010-2011 for Natural Sciences and Engineering Research Council (NSERC) grants and scholarships; from $8.7 million in 1999-2000 to $24.8 million in 2013-2014 for Canadian Institutes of Health Research (CIHR) grants and awards; from $22.9 milion in 2001-2002 to $43.0 million in 2012-2013 for Social Sciences and Humanities Research Council of Canada (SSHRC) funding; and from $0 to $33.7 million from the Networks of Centres of Excellence (NCE) to support three centres of excellence over four to five years. However, Atlantic Canada’s share of national funding has remained at about 5% of total NSERC expenditures, 3% of total CIHR funding in Canada; and 6.5% of total SSHRC expenditures in Canada.

[xiii] Statistics Canada, CANSIM table 358- 0001. Gross domestic expenditures on research and development, by science type and by funder and performer sector (current prices).

[xiv] NSERC: Natural Sciences and Engineering Research Council; NRC-IRAP: National Research Council of Canada Industrial Research Assistance Program; CIHR: Canadian Institutes of Health Research; NCE: Networks of Centres of Excellence; CFI: Canada Foundation for Innovation; BDC: Business Development Bank of Canada.

[xv] Currently called Department of Business, Tourism, Culture and Rural Development.

[xvi] Mobilizing Science and Technology to Canada’s Advantage, 2007

[xvii] Government of Canada, Budget 2014. http://www.budget.canada.ca/2014/docs/plan/ch3-2-eng.html.

[xviii] Government of Canada, Speech from the Throne to open the Second Session of the Forty-first Parliament of Canada, October 16, 2013. House of Commons Journals: 2013.10.16, vol. 147. https://lop.parl.ca/sites/ParlInfo/default/en_CA/Parliament/procedure/throneSpeech/speech412.

[xix] Government of Canada, Speech from the Throne to open the First Session of the Forty-first Parliament of Canada. June 3, 2011. House of Commons Journals: 2011.06.03, vol. 146.

[xx] Industry Canada, Innovation Canada: A Call to Action. Review of Federal Support to Research and Development – Expert Panel Report. http://publications.gc.ca/site/eng/404471/publication.html

[xxi] Government of Canada, Constitution Act, 1867 to 1982, Part III, Equalization and Regional Disparities, section 36. (1) (b). http://laws- lois.justice.gc.ca/eng/Const/page-16.html#h-53

[xxii] Government of Canada, Atlantic Canada Opportunities Agency Act, [Enacted as Part I to R.S.C. 1985, c. 41 (4th Supp.), in force September 15, 1988. https://laws-lois.justice.gc.ca/eng/Const/page-16.html#h-53

[xxiii] Speech from the Throne, October 2013

[xxiv] The $400-million Venture Capital Action Plan, announced in January 2013, will establish up to two new large-scale private-sector-led national funds with $250 million. The Canada Accelerator & Incubator Program, to be administered by NRC-IRAP, will help establish a critical mass of incubators and accelerators. The Technology Demonstration Program provides non-repayable contributions in support of large-scale technology demonstration projects in the aerospace, defence, space and security sectors.

[xxv] The Federal Economic Development Agency for Southern Ontario, the Federal Economic Development Initiative for Northern Ontario, Canada Economic Development for Quebec Regions, and Western Economic Diversification Canada.

[xxvi] This total does not include clients who undertook one or more AIF projects under the IC sub-program and one or more BDP projects, which were all coded to the IBD or PG sub- programs.

[xxvii] From 2005-2006 to 2012-2013, average annual assistance approved by ACOA under the IC sub-program was approximately $26.2 million for AIF commercial projects, $25.6 million for AIF non-commercial projects, $25.6 million for BDP commercial projects and $5.0 million for BDP non-commercial projects, including Community Adjustment Fund (total $82.4 million).

[xxviii] The average ratings are summarized in Appendix E.

[xxix] Statistics Canada, CANSIM table 358- 0001. Gross domestic expenditures on research and development, by science type and by funder and performer sector (current prices).

[xxx] Note that “institutional” includes universities, colleges and research institutions; associations and public-sector organizations are excluded from this analysis.

[xxxi] The average ratings provided by clients are summarized in Appendix E.

[xxxii] Detailed ratings are provided in Appendix F.

[xxxiii] The average is calculated including those who generated no revenues.

[xxxiv] Pharmaceutical Research and Manufacturers of America, Biopharmaceutical Research Industry: 2013 Profile.

[xxxv] The financing does not involve the sale of a company's shares, and therefore does not dilute the ownership of existing shareholders.

[xxxvi] Detailed ratings are provided in Appendix E.

[xxxvii] Atlantic Canada Opportunities Agency, “Harper Government Unveils Major New Commitment to Supporting Innovation and Commercialization in Atlantic Canada,” July 16, 2014. http://www.acoa- apeca.gc.ca/eng/Agency/mediaroom/NewsReleases/Pages/4346.aspx.

[xxxviii] National Research Council of Canada, 2012-13 Departmental Performance Report – Supplementary Tables. http://www.nrc- cnrc.gc.ca/eng/reports/2012_2013/dpr_2013/dpr_table_01.html.

[xxxix] National Research Council of Canada, Annual Report 2012-2013. http://ww w.nrc- cnrc.gc.ca/eng/about/planning_reporting/annual/2012_2013/discussion_analysis.html#fsda2_2.

[xl] Natural Sciences and Engineering Research Council, 2012-13 Departmental Performance Report: Details on Transfer Payment Programs. http://www.nserc-crsng.gc.ca/NSERC-CRSNG/Reports-Rapports/plans- plans_eng.asp.

[xli] Natural Sciences and Engineering Research Council: Idea to Innovation Grants. http://www.nserc-crsng.gc.ca/Professors-Professeurs/RPP- PP/I2I-INNOV_eng.asp.

[xlii] Natural Sciences and Engineering Research Council: Collaborative Research and Development Grants. http://www.nserc-crsng.gc.ca/Professors- Professeurs/RPP-PP/CRD-RDC_eng.asp.

[xliii] Canadian Institutes of Health Research, 2012-2013 Departmental Performance Report – Supplementary Tables. http://www.cihr-irsc.gc.ca/e/47335.html.

[xliv] Social Sciences and Humanities Research Council, Funding by Region, Province and Institution for 2012-2013.

[xlv] Networks of Centres of Excellence of Canada: Funded Networks and Centres. http://www.nce-rce.gc.ca/NetworksCentres- CentresReseaux/Index_eng.asp.

[xlvi] Canada Foundation for Innovation: Projects Funded. http://www.innovation.ca/en/OurInve stments/Projectsfunded.

[xlvii] Business Development Bank of Canada, 2013 Annual Report. http://w ww.bdc.ca/EN/about/corporate_governance/financial_results/Pages/default.aspx.

[xlviii] Canada Revenue Agency: Who can claim SR&ED tax incentives and what are the benefits? http://www.cra-arc.gc.ca/txcrdt/sred- rsde/clmng/whcnclmsrd-eng.html.

[xlix] Industry Canada: TDP vs. SADI. http://www.ic.gc.ca/eic/site/ito- oti.nsf/eng/h_00860.html.

[l] Mitacs. http://www.mitacs.ca/.

[li] Government of Canada, Venture Capital Action Plan. http://www.fin.gc.ca/vcap-pacr/index-eng.asp.

[lii] Government of Canada, National Shipbuilding Procurement Strategy. http://www.tpsgc-pwgsc.gc.ca/app-acq/sam-mps/snacn-nsps-eng.html.

[liii] Sustainable Development Technology Canada. http://www.sdtc.ca/index.php? page=home.

[liv] Innovacorp, 2012-2013 Accountability Report. https://innovacorp.ca/about-us/corporate-publications.

[lv] New Brunswick Innovation Foundation. http://nbif.ca/en/about_nbif/.

[lvi] Nova Scotia Department of Economic and Rural Development and Tourism: Innovation and Learning. http://www.novascotia.ca/econ/ial/.

[lvii] Innovation PEI: Programs and Services. http://www.innovationpei.com/Programs-Services.

[lviii] Innovation PEI, 2012-2013 Annual Report. http://www.innovationpei.com/resources.

[lix] Research & Development Corporation of Newfoundland and Labrador, 2012-2013 Annual Report. http://www.rdc.org/%5C/about/index.htm.

[lx] Research & Development Corporation of Newfoundland and Labrador. http://www.rdc.org/%5C/funding/index.htm.

[lxi] New Brunswick Department of Economic Development. http://www2.gnb.ca/co ntent/gnb/en/departments/economic_development.html.

[lxii] Regional Development Corporation. http://www2.gnb.ca/co ntent/gnb/en/departments/regional_development.html.

[lxiii] RPC. http://www.rpc.ca/english/index.html.

[lxiv] Springboard. http://springboardatlantic.ca/about-us/.

[lxv] BioNova. http://bionova.ca/about/.

[lxvi] BioNB. http://bionb.org/en.

[lxvii] The Next Phase. http://thenextphase.ca/.

[lxviii] Propel ICT/Launch 36. http://launch36.ca/.

[lxix] Volta – Halifax Startup House. http://www.voltaeffect.com/.

[lxx] Venn Innovation Inc. http://venncentre.ca/.

[lxxi] Wallace McCain Institute. http://www.wallacemccaininstitute.com/.

[lxxii] Pond-Deshpande Centre for Innovation & Entrepreneurship. http://www.ponddeshpande.ca/.

[lxxiii] Genesis Centre. http://www.genesis.mun.ca/GenesisCentre/.

Page details

Date modified:: 2020-04-23

Evaluation of ACOA’s innovation and commercialization sub-program - final report

Glossary of acronyms

Table of contents

List of figures

List of tables

Acknowledgements

Executive summary

1.0 Introduction

1.1 Evaluation overview

1.2 Evaluation approach and methodology

1.3 Evaluation challenges and mitigation measures

2.0 Profile of the innovation and commercialization sub-program

2.1 Sub-program profile

2.2 Sub-program logic model

2.3 Sub-program expenditures

2.4 Governance and management

3.0 Findings: relevance

3.1 Continued need for the programming

3.2 Relationship to other programming

3.3 Alignment with federal priorities, roles and responsibilities

4.0 Findings: performance – effectiveness

4.1 Investment in innovation and commercialization projects

4.2 Achievement of expected outcomes

4.2.1 Immediate outcomes

4.2.2 Intermediate outcomes

4.2.3 Long-term outcomes

4.3 Contributing and constraining factors and lessons learned

4.4 Adequacy of performance measurement

5.0 Findings: performance – efficiency and economy

5.1 Efficient utilization of resources

5.2 Structures and mechanisms that support efficiency and economy

5.3 Delivery costs

5.4 Alternative modes of delivery

6.0 Conclusions and recommendations

6.1 Conclusions

6.2 Recommendations

Appendix A: management action plan

Appendix B: overview of ACOA

Appendix C: evaluation issues and questions

Appendix D: innovation and commercialization sub-program logic model

Appendix E: other programs supporting innovation and commercialization in Atlantic Canada

Appendix F: client survey ratings tables

Appendix G: profile of IC projects, by program and type of client, 2005-2006 to 2012-2013

Footnotes

Page details