Evaluation of the Atlantic Canada Opportunities Agency’s International Business Development Sub-Program
FINAL REPORT

Evaluation Unit
Evaluation and Risk Directorate
Atlantic Canada Opportunities Agency

May 22, 2015

Glossary of Acronyms

ACOA Atlantic Canada Opportunities Agency

BDP Business Development Program

CD Community Development

DFATD Department of Foreign Affairs and Trade and Development

DG director general

DPR Departmental Performance Report

EAP Economic Action Plan

ED Enterprise Development

FDI foreign direct investment

FTA free trade agreement

FTE full-time equivalent

G&Cs grants and contributions

GMAP Global Market Action Plan

IBD International Business Development (an ACOA sub-program)

IBDA Canada-Atlantic Provinces Agreement on International Business Development / International Business Development Agreement

IBDP International Business Development Program

IC Innovation and Commercialization

KI key informant

MAP management action plan

O&M operations and maintenance

PAA program alignment architecture

PAC Policy, Advocacy and Coordination

PG Productivity and Growth

PMS performance measurement strategy

SMEs small and medium-sized enterprises

TB Treasury Board

TI Trade and Investment

Table of Contents

Acknowledgements

Executive Summary

1.0 Introduction

1.1 Evaluation Overview

1.2 Evaluation Design and Methodology

1.3 Evaluation Strengths, Limitations and Mitigating Measures

2.0 Profile of the International Business Development Sub-program

2.1 Context

2.1.1 International Business Development Sub-program

2.2 Program Theory

2.3 Program Accountability and Governance

2.3.1 Project Development and Signing Authority

2.4 Expenditure Profile

2.5 Project Profile

3.0 Findings: Relevance

3.1 Continued Need for the Programming

3.2 Alignment with Government Priorities

3.3 Alignment with Federal Roles and Responsibilities

4.0 Findings: Performance – Effectiveness

4.1 Incrementality

4.2 Achievement of Expected Outcomes

4.2.1 Immediate Outcomes

4.2.2 Achievement of Intermediate Outcomes

4.2.3 Long-Term Outcomes

4.3 Unintended Outcomes of IBD Programming

4.4 Barriers and Facilitators to the Achievement of IBD Outcomes

4.5 Adequacy of Performance Measurement

5.0 Findings: Performance – Efficiency and Economy

5.1 Efficient Utilization of Resources

5.2 Structures and Mechanism that Support Efficiency and Economy

5.3 Alternative Modes of Delivery

6.0 Conclusions and Recommendations

6.1 Conclusions

6.2 Recommendations

Appendix A – IBD Evaluation Management Action Plan

Appendix B – Evaluation Questions, Judgment Criteria and Methods

Appendix C – IBDA Evaluation Findings

Appendix D – Program Alignment Architecture

Appendix E – IBD Conceptual/Analytical Framework

Footnotes

List of Figures

Figure 1: Logic Model for International Business Development Sub-program

Figure 2: Percentage of Approved IBD Funding by Sector, 2008-2009 to 2013-2014

Figure 3: IBD Clients Accessing Other ACOA Funding Programs, 2008-2009 to 2013-2014

List of Tables

Table 1: IBD Evaluation Questions by Core Issue

Table 2: IBD Sub-program Expenditures ($M), 2008-2009 to 2013-2014

Table 3: Approved IBD Projects and Funding ($M), 2008-2009 to 2013-2014

Table 4: Number of Projects by Primary Type and Funding ($M), 2008-2009 to 2013-2014

Table 5: Summary of Organizations Delivering IBD Programming in Atlantic Canada

Table 6: Funds Leveraged by ACOA, by Program Funding, from 2008-2009 to 2013-2014

Table 7: Alignment of IBD Evaluation Findings, Conclusions and Recommendations

Acknowledgements

This evaluation provides the Atlantic Canada Opportunities Agency’s (ACOA) management with systematic, neutral evidence on the relevance and performance of its International Business Development sub-program. The results of this evaluation will be used to support the future direction of the programming.

The study was managed and implemented by ACOA’s Evaluation Unit, with advice and support from the members of the Evaluation Advisory Committee. Members of the committee played an important role throughout the evaluation in ensuring that the study and its recommendations are both relevant and useful. Special thanks go to four external committee members:

We are also grateful to the many ACOA staff members, including members of the Evaluation Unit, working group members, ACOA clients and external key informants, who provided their time and knowledge in support of this study.

All of these contributions were instrumental in understanding the outcomes, best practices and areas for improvement for ACOA’s International Business Development programming.

Courtney Amo
Director, Evaluation and Risk Directorate (Head of Evaluation)
Atlantic Canada Opportunities Agency

Executive Summary

This report presents the findings, conclusions and recommendations of the evaluation of ACOA’s International Business Development (IBD) sub-program. The evaluation examines the relevance and performance of IBD programming from 2008-2009 to 2013-2014 as per the Agency’s approved evaluation plan and the evaluation’s terms of reference. It meets the accountability requirements outlined in the Treasury Board’s (TB) Policy on Evaluation.[i] The study addresses specific needs of senior management and the renewal of the Canada-Atlantic Provinces Agreement on International Business Development (IBDA).

The IBD sub-program supports small and medium-sized enterprises (SMEs), not-for-profit organizations, provincial governments, and research and educational institutions with a view to helping Atlantic Canadians strengthen their position in international markets and take advantage of international business opportunities. Through trade missions, marketing, training, networking and planning supports, the programming seeks to increase the number of new exporters while also working with existing ones to expand export activities in established and emerging markets.

Relevance

The findings of the evaluation indicate that there is a strong and continued need for the IBD programming. Challenges to international business development persist and, to some extent, have increased since the 2010 Trade and Investment evaluation. ACOA is aware of these challenges and has adapted its programming to address the needs of clients by focusing on emerging markets and helping SMEs benefit from free trade agreements (FTAs).

International business development activities are aligned with ACOA’s strategic outcome and the ACOA Act. They are consistent with the Agency’s roles and responsibilities and broader federal priorities on increasing trade and attracting investment. While recruiting and retaining foreign students is a priority of the federal government, an opportunity exists to better articulate the alignment of these activities within the IBD sub-program. Likewise, the attraction of foreign direct investments (FDI) is clearly aligned with federal roles and priorities though ACOA’s approach and level of involvement, and the types of activities undertaken and supported varies across regional offices.

The Agency’s IBD programming complements rather than duplicates programming being delivered by other organizations. Specific characteristics that set the programming apart include a focus on Atlantic Canada and on SMEs; attention to economic development outcomes; and meeting federal government priorities. IBD programming provides important financing not available from other funders. It is also complementary to other sub-programming offered by ACOA, particularly to Productivity and Growth (PG) and Innovation and Commercialization (IC) programming.

Effectiveness: Achievement of Expected Outcomes

The evaluation findings indicate that the Agency’s IBD activities are incremental to the achievement of immediate, intermediate and ultimate outcomes. The programming achieves positive immediate outcomes, including improved awareness of international business opportunities and increased capacity among SMEs as well as strengthened partnerships and alliances. To a lesser extent, the evaluation is able to show that the sub-program contributed to the achievement of intermediate outcomes of more first-time exporters, increased foreign direct investments, and increased acquisition and commercialization of new technology.

These immediate and intermediate outcomes contribute to the ultimate IBD sub-program expected outcome of “increased economic benefits for Atlantic Canada from international markets,” as well as the Enterprise Development (ED) program outcome of “improved growth and competitiveness for Atlantic SMEs.” There were challenges to comparing the results of this evaluation with those of the 2010 Trade and Investment sub-program evaluation due to changes in program alignment architecture (PAA) and performance measurement strategy (PMS) as well as differences in the evaluation scope, design and approaches.

Some factors continue to impede the success of SMEs in international markets, including SME capacity and delays in reaching markets. This is specifically mitigated through the Agency’s support for skills development as well as knowledge acquisition. Key informants and survey results also identified internal communication and coordination, timeliness of project approvals, and financial constraints as barriers to project implementation and success.

While improvements have been made to ensure availability and use of performance measurement information, there is a need to address specific gaps in data. These gaps include: information on intermediate and longer-term outcomes and the results of non-mission activities, including those related to FDI and the education sector; unavailability of Statistics Canada data comparing IBD-assisted SMEs with non-clients; and cumbersome data collection tools and processes.

Effectiveness: Efficiency and Economy

The IBD programming demonstrates efficiency and economy in the utilization of resources.[ii] The cost of delivering IBD programming has remained relatively stable since the previous evaluation. While the cost of delivering IBD activities is higher than for other ACOA programming, the evaluation recognizes that this is related to aspects of the delivery model that are important to the achievement of expected outcomes.

Mechanisms are in place to ensure efficient use of resources, including governance structures, planning and budgeting processes and a decentralized delivery model. One area where efficiency and economy could be enhanced is through better articulation of Agency priorities, activities and expected outcomes of IBD programming, as some weaknesses related to planning, internal communication and coordination pose risks to the strategic achievement of outcomes.

Conclusions

Overall, the evaluation found that ACOA’s IBD sub-program continues to be relevant and effective. In summary:

Recommendations

The evaluation identifies three recommendations related to: improving IBD programming by balancing the need for a consistent, targeted approach with the need to allow for flexibility to address pan-Atlantic and provincial realities, assets and gaps; ensuring that the governance of IBD programming allows for shared planning and clearer decision making; and improving the availability and access of information and knowledge to enhance results management.

Recommendation 1: ACOA should build on its corporate knowledge and current best practices in the IBD to develop and communicate a corporate IBD strategy. A clearly articulated and communicated strategy for IBD programming should:

Recommendation 2: To ensure that the governance framework supports evolving interests and needs, including those that emerge from the development of an IBD strategy (Recommendation 1), ACOA should review the governance structures for IBD programming. The governance review should:

Recommendation 3: To support results-based management, ACOA should build on its previous efforts to improve the IBD performance measurement strategy by reviewing and updating the program’s indicators and expected outcomes, and revising tools and processes. This should include:

The management action plan (MAP) prepared by IBD management to address each of the evaluation’s recommendations is presented in Appendix A. The full evaluation question framework, including judgment criteria, methodology and level of effort, is presented in Appendix B.

1.0 Introduction

This report presents the findings, conclusions and recommendations of the evaluation of the Agency’s IBD sub-program, which aims to increase economic benefits for Atlantic Canada from international markets. The activities that currently fall under the IBD sub-program were last evaluated as part of the former Trade and Investment (TI) program sub-activities evaluation, approved in November 2010.

This evaluation examines the relevance and performance of IBD programming as per the Agency’s approved evaluation plan and the evaluation’s terms of reference. The evaluation meets the accountability requirements outlined in the TB’s Policy on Evaluation.[iii] It also addresses specific needs of senior management and the renewal of the IBDA. The scope of the evaluation included IBD activities, those of the IBDA as well as those managed by the former Enterprise Cape Breton Corporation[iv] funded between 2008-2009 and 2013-2014.

The evaluation was led by an independent evaluation team from ACOA’s Evaluation and Risk Directorate. An expert consulting firm provided support for the administration of the client survey. An Evaluation Advisory Committee chaired by the Director, Evaluation and Risk (Head of Evaluation) supported the evaluation. The committee included representation from ACOA as well as external stakeholders.

Sections 1 and 2 of this report provide an overview of the evaluation approach and a profile of the IBD sub-program. Sections 3 to 5 present the evaluation study’s findings by broad area of relevance, performance-effectiveness, and performance-efficiency and economy. Section 6 presents the conclusions and recommendations. The MAP prepared by IBD management to address each of the evaluation’s recommendations is presented in Appendix A.

1.1 Evaluation Overview

This evaluation provides timely, credible and neutral information on the relevance and performance of IBD programming to support decision making, continuous improvement and results-based management. In addition to the TB Policy on Evaluation requirements, the planning and calibration of the study was influenced by three initiatives: an extensive ED sub-program evaluation planning study; an internal IBD review; and reporting needs related to the renewal of the IBDA.

An extensive planning study was completed in 2012 to clarify the objectives, scope and utilization of the three ED sub-program evaluations. It helped gauge evaluation readiness by testing the program logic/theory, building program profiles, clarifying data requirements, and testing data availability and quality prior to the evaluation being conducted.

This evaluation addresses particular questions that were outstanding from the internal review of IBD programming led by head office IBD program management with support from the Policy unit. The review, which began in 2012-2013, examined the economic context, federal policies and other contextual factors in order to set the future direction of the programming. The evaluation used the information collected through the review to calibrate the document and literature review methodology for this study.

The evaluation also considered the needs of senior management and key stakeholders as they relate to the planned renewal of the IBDA, which is set to expire on March 31, 2016. Separate evaluation questions were developed to ensure the IBDA management committee would have clear data to support the renewal of the agreement. Detailed information on the IBDA-specific questions is found in the overall IBD sub-program evaluation framework, Appendix B, and detailed IBDA findings are found in Appendix C.

1.2 Evaluation Design and Methodology

As required by the TB Policy on Evaluation (2009), the evaluation team used a risk-based approach for the design of the IBD evaluation. Based on the results of a document review and stakeholder consultations, the level of effort associated with conducting the evaluation was calibrated to reflect risks associated with the relevance and effectiveness of the programming.

The evaluation was also calibrated to meet the information and timing needs of ACOA’s senior management and of the IBDA management committee. It maximized usage of information generated from an extensive ED planning study, including an IBD impact framework, and the IBD review. Time and other resources were saved by using previously developed survey questionnaires from other Agency sub-program evaluations. Interviews were strategically conducted to reduce the burden on clients and ACOA staff and to fill information gaps where necessary.

The evaluation of the IBD sub-program addressed five core issues that fall under the two broad categories of relevance and performance, in accordance with the TB Policy on Evaluation. Table 1 identifies the specific evaluation questions for each core issue as per the terms of reference for the study approved by the President.

Table 1: IBD Evaluation Questions by Core Issue

Issue

Evaluation Question

Relevance

Issue 1: Continued Need for the Program

1.1

To what extent does the IBD sub-program continue to address a demonstrable need?

1.2

To what extent is the IBD sub-program responsive to existing and emerging needs of Atlantic Canadians?

Issue 2: Alignment with Government Priorities

2.1

To what extent is the IBD sub-program aligned with federal government priorities and expectations, and to ACOA’s strategic outcome?

Issue 3: Alignment with Federal Roles and Responsibilities

3.1

To what extent does the IBD sub-program align with federal roles and responsibilities? What other mechanisms exist to address these needs?

3.2

To what extent does the IBD sub-program activities, including the attraction of FDI, duplicate, overlap or complement other programming, including other ACOA sub-programs?

Performance

Issue 4: Effectiveness

4.1

Incrementality: What impact would the absence of IBD programming have on expected program outcomes?

4.2

How and to what extent has the IBD sub-program contributed to immediate, intermediate and long-term outcomes?

4.3

What if any unintended outcomes were achieved through IBD programming?

4.4

What are the facilitators/best practices and barriers/lessons learned to achieve the IBD’s immediate, intermediate and ultimate outcomes, and to what extent are these being mitigated?

4.5

To what extent have the IBD performance measurement and reporting structures been implemented and effectively contributed to the reporting of IBD outcomes? How is the performance information being used by ACOA to support decision making?

Issue 5: Efficiency and Economy

5.1

To what extent are delivery costs reasonable in relation to outcomes achieved?

5.2

To what extent are planning, performance measurement, budgeting, governance and delivery mechanisms in place that contribute to the achievement of IBD outcomes in the most efficient and economical manner?

5.3

To what extent has program management considered and implemented alternative modes of delivery for the IBD sub-program?

Judgment criteria or “benchmarks” for successful achievement for each evaluation question were developed with advice from program management. These criteria were considered in determining data needs, methodology and analysis. They were used by the evaluation team to objectively and transparently assess the relevance and performance of the programming.

In consideration of the evaluation issues, risks and judgment criteria, the evaluation team chose a mixed-methods research design involving multiple lines of evidence. A mixed-methods approach allowed for convergence of results across lines of evidence while developing better understanding by exploring different aspects of an evaluation issue.[v] Detailed method notes were developed for each method, with the advice of Evaluation Advisory Committee members and guided study implementation. The following methods were used for the IBD evaluation.

Document and literature review

The document review built upon efforts undertaken during the development of the IBD impact analysis framework as well in the conduct of the IBD review. The focused literature review was targeted at filling gaps as they pertain to the evaluation questions. Documents reviewed included project-related materials, parliamentary reports (e.g. Report on Plans and Priorities), presentations to senior management, policy research studies, published literature, federal strategy documents, and relevant TB submissions.

Program data review

Existing program data analyzed during the evaluation included project data, performance measurement data and operational data.

Key informant interviews

Interviews were conducted strategically in order to address knowledge gaps and to validate understanding of the programming. A total of 58 key informants (KIs) were interviewed. To maximize participation of interviewees, group interviews were undertaken with some internal stakeholders. The participants included a cross section of internal ACOA management and staff (46) as well as external stakeholders (12) from provincial governments, other federal government departments, industry associations, SMEs and a university.

In order to address senior management questions, many of which concerned internal issues such as ACOA roles, program delivery and governance, the evaluation team chose to focus the majority of KI interviews on internal stakeholders. The higher proportion of ACOA staff and managers was balanced by analysis of other lines of evidence.

Client survey

Survey results were mostly used to inform evaluation findings related to the achievement of outcomes. The IBD client survey was developed in conjunction with surveys for other ED sub-programs, which not only created consistency in the types of questions asked but also resulted in substantial time and resource savings.

Clients who had multiple IBD projects over the period of evaluation were only surveyed once, and those who had recently completed a survey as part of the PG evaluation were removed from the population. Of 263 eligible clients, 202 IBD clients were sampled and contacted for the survey and 121 clients completed the survey. All 19 unique IBDA clients in the sample completed the survey; the remaining 102 were IBD clients. Based on a confidence level of 95 per cent, 121 clients completed or substantially completed the survey, which met the targeted response rate of 50 per cent with a margin of error of about ±6.5 per cent. The clients who completed the survey were representative of the full population of IBD projects over the period, including distribution across the four ACOA regional offices and those of pan-Atlantic scope as well as across type of programming assistance.

1.3 Evaluation Strengths, Limitations and Mitigating Measures

The design and implementation of the evaluation is appropriate given the objectives of the study. The evaluation design adequately addressed limitations through appropriate mitigation strategies resulting in findings, conclusions and recommendations that are reliable and valid.

The IBD evaluation design was guided by program theory and incorporated multiple methods to better understand the complex factors that influence relevance and the achievement of outcomes. The evaluation design incorporates comparison to the 2010 evaluation of TI program sub-activities where feasible and appropriate.

A strength of the evaluation lies in how it was calibrated to meet specific information needs within shortened timelines. In particular, the evaluation scope and timing were adjusted to meet the needs of the internal IBD review and the renewal of the IBDA. With advice from the Evaluation Advisory Committee, the evaluation team calibrated the scope, methodology, data collection and governance for the study. Calibration was supported by maximizing use of the findings of the IBD review and other existing data as well as strong coordination among the evaluation teams conducting concurrent ED-related evaluations, including the use of one consulting firm for the three ED sub-program client surveys.

The evaluation team engaged key stakeholders throughout the study. There was strategic engagement of senior management and members of the IBDA management committee in the identification of key evaluation questions, development of methods, completion of head office and regional office delivery profiles, identification of KIs and discussion of preliminary findings and conclusions and recommendations.

The evaluation acknowledges the challenges that affected the study and mitigation strategies were implemented where possible. Similar to other ACOA studies, a variety of external factors affect the achievement of IBD expected outcomes, and a significant amount of time is generally needed to achieve intermediate and long-term outcomes. The evaluation assessed the contribution of the Agency`s programming to the achievement of outcomes, especially at the intermediate and long-term levels using the IBD conceptual/analytical framework, developed during the planning of all the ED programming evaluations in 2012-2013.

There were changes to performance measurement indicators and data collection during the evaluation period that effected the assessment of results. In particular, data comparing SMEs that received ACOA assistance with comparable firms was not consistently available. In 2010-2011 and 2011-2012, Statistics Canada provided data to support findings for “percentage points by which the growth in export sales of ACOA-assisted firms exceeds that of unassisted firms.” Statistics Canada and ACOA staff identified changes in the methodology used by Statistics Canada that caused issues with data comparison starting with the 2012-2013 Departmental Performance Report (DPR). As a result, the comparison of export sales between assisted and non-assisted firms is no longer being used as an IBD performance target.

Another limitation of this study was the inability to compare all client survey results to those of the previous evaluation period. This was due in part to differences in the design of the two surveys, changes to the PAA and the calibration of the study to assess particular questions.

Overall, the diversity of methods helped compensate for the inherent limitations of each data source and helped mitigate the overall study challenges. Multiple lines of evidence gathered through a mix of qualitative and quantitative methods allowed for the convergence of results across methods, and a better understanding of outcomes was gained by exploring different facets of the complex issues associated with international business development.

2.0 Profile of the International Business Development Sub-program

2.1 Context

Focusing on expanding trade and attracting investment to Atlantic Canada, the IBD sub-program falls within the ED program in the Agency’s PAA (Appendix D). In addition to the IBD, ED programming is delivered through two other sub-programs: IC and PG. The three ED sub-programs aim to help the Atlantic region become innovative and productive and to improve the growth of the economy. The ED programming works in collaboration with the Agency’s other three broad program areas – CD, PAC and Internal Services[vi] – to support the Agency’s strategic outcome of developing “a competitive Atlantic Canadian economy.”

2.1.1 International Business Development Sub-program

The IBD sub-program supports SMEs, not-for-profit organizations, provincial governments and research and educational institutions with a view to helping Atlantic Canadians strengthen their position in international markets and take advantage of international business opportunities. Through trade missions, marketing, training, networking and planning supports, the programming seeks to increase the number of new exporters while also working with existing exporters to expand activities in established and emerging markets.

The core activity of IBD programming is trade missions, primarily outgoing but also the hosting of incoming trade delegations. Trade missions help SMEs develop knowledge and skills, gather market intelligence, identify new contacts and business leads, and create partnerships and alliances. The creation of partnerships and alliances helps Atlantic Canadian businesses acquire technology, commercialize their products and services, diversify markets and enter global value chains.

Trade missions generally follow a structured approach that includes business briefings, plenary/sector sessions, site visits and networking events. Mission participants are introduced to potential partners and have pre-arranged business-to-business meetings tailored to their individual needs. Marketing activities are often part of trade missions and help promote Atlantic Canada’s specific capabilities and products within international markets.

In addition to capacity development that occurs through trade missions, the IBD sub-program develops SME knowledge and skills for international business by supporting market and sector awareness events and other training activities. IBD programming also uses O&M funding to help businesses identify opportunities, implement strategies and develop marketing tools to improve international business success.

An important role of the IBD sub-program is to coordinate the efforts of federal, provincial and private-sector organizations in Atlantic Canada. This is undertaken through regional office delivery of IBD programming that coordinates closely with individual provincial stakeholders. Head office also coordinates programming with the efforts of a variety of pan-Atlantic IBD stakeholders through the IBDA, a pan-Atlantic initiative involving ACOA, the four Atlantic provincial governments, the Department of Foreign Affairs, Trade and Development (DFATD) and Industry Canada. While the IBDA funds only non-commercial projects, the primary objectives of the IBDA align with those of the rest of IBD programming[vii].

IBDA projects are developed by 12 priority sector working groups and four market working groups that include federal, provincial, industry and other stakeholders. The sector working groups are:

The four market working groups focus on emerging markets in Brazil, India and China as well as the European Union to build on opportunities related to the Canada-European Union Trade Agreement. The IBDA also supports projects in the United States, the key established market for Atlantic Canada, as well as projects seeking to expand in emerging markets such as Asia Pacific, Latin America and the Caribbean.

2.2 Program Theory

A program’s theory serves to communicate the assumed causal connections between program elements. The theory behind the IBD sub-program is illustrated in its logic model (Figure 1), which shows the reach, activities and expected outcomes of the programming.[viii] The key long-term expected outcome of the IBD sub-program is increased economic benefits for Atlantic Canada from international markets.

Through international business missions, marketing, research, market intelligence and other activities, the IBD sub-program is expected to result in improved awareness of international business opportunities and increased capacity to engage in international business activities among its client groups. These outcomes are expected to lead to greater foreign investment opportunities and increased export activity by SMEs in Atlantic Canada. The IBD is also expected to contribute to increased acquisition and commercialization of new technology, this outcome is reported on under the IC and PG sub-programs.

Figure 1: Logic Model for International Business Development Sub-program

Reach

Clients / Direct Target

SME’s

Educational and Research Institutions

Not-for profit organizations
Provinces

Co-deliverers

DFATD

Provinces

Associations

Activities and Outputs

Research and market intelligence
Outreach activities
Domestic communications
Learning and skill development opportunities
International initiatives
Foreign communications
Financial support to clients
Pursuit of international business opportunities

Immediate / Direct Outcomes

Improved Atlantic Canadian awareness of international business opportunities

Increased international business capacity for SMEs, universities and research establishments in Atlantic Canada

Intermediate / Indirect Outcomes

Expanded foreign investment opportunities (FDI and Canadian direct investment abroad)

Increased export activity by SMEs in Atlantic Canada

Increased acquisition and commercialization of new technology (reported on under IC and PG sub-programs)

Long-term Outcome

Increased economic benefits for Atlantic Canada from international markets

Enterprise Development

Improved growth and competitiveness of Atlantic SMEs

Strategic Outcome

A competitive Atlantic Canadian economy

Source: ACOA 2012-2013 Program Activities and Sub-activities Logic Models. February 2012.

In addition to the IBD logic model, an impact conceptual framework (Appendix E) was drafted in 2012 as part of an evaluation planning process for all ED sub-programs. The framework illustrates the types of activities undertaken, the stakeholders involved, the immediate and intermediate outcomes, and the desired impacts. It shows the complex environment in which IBD programming operates, including both ACOA’s contribution via other sub-programs and external contextual factors. The framework shows the interrelationships between IBD activities, outcomes and impacts as well as the contributions of other stakeholders.

2.3 Program Accountability and Governance

ACOA’s executive committee is the highest internal governing forum that supports the president (ACOA’s deputy minister) in the development of the policies and programs of the Agency. Members include regional vice-presidents and related senior executives as identified by the president to ensure the appropriate representation of expertise and experience.

Responsibility for the ED program activity and related sub-programs, including the IBD, is shared between ACOA’s head office, located in Moncton, N.B., and its regional offices. At the head office level – and unlike the other two ED sub-programs, which fall under the responsibility of the director general (DG) of ED, the IBD sub-program is led by a DG of IBD. Both DGs report to the vice-president, Policy and Programs, who in turns reports to the president of ACOA.

ACOA’s regional offices are located in Halifax, Charlottetown, St. John’s and Fredericton. Each regional office, led by a vice-president, is responsible for contributing to the development of ACOA policies and the delivery of Agency programs within their own province.[ix] IBD programming falls under the responsibility of regional directors of ED, who report to regional DGs of operations. The PEI region is the only one where IBD programming is led by a director of IBD who reports to the regional DG Operations.

Under this governance structure, regional ED officers are generally responsible for the day-to-day management of contribution agreements between recipients and the Agency. Regional ACOA staff work closely with provincial governments and other stakeholders to meet specific needs and support key sectors. ACOA’s regional delivery offices fund both commercial and non-commercial projects. Head office programming focuses on non-commercial, pan-Atlantic initiatives, including the IBDA. Similar to regional office activities, pan-Atlantic initiatives are coordinated with provincial governments, industry associations, universities and other stakeholders.

An IBDA management committee comprised of ACOA, DFATD, Industry Canada and the four provincial governments establishes IBDA procedures, approves and evaluates projects, appoints sub-committees and reports results. The Government of Canada, represented by ACOA as the lead federal agency, provides 70 per cent of the funding, with the remaining 30 per cent provided by the provinces as follows: New Brunswick (11 per cent); Nova Scotia (11 per cent); Newfoundland and Labrador (5 per cent); and Prince Edward Island (3 per cent). The provincial partners transfer their contributions to ACOA on an annual basis for disbursement to approved projects.

Agriculture and Agri-Food Canada participates on the IBDA management committee as an observer. The committee is supported by a secretariat housed at ACOA’s head office and manages the day-to-day operations and administration of the IBDA, including the issuance of contracts, payments, reporting and surveys.

2.3.1 Project Development and Signing Authority

Project development and signing authority procedures differ by funding program and by regional office. The Business Development Program (BDP) is the main transfer payment program used to fund projects under the IBD sub-program, including IBDA projects.[x]

BDP signing authority is delegated to various levels depending on the office and type of project and is also established per the Agency’s financial signing authority delegation instrument. Across ACOA’s offices, the level of signing authority for a BDP contribution ranges from no signing authority to up to $100,000 for account managers, up to $300,000 for the DGs of IBD; and up to $500,000 for regional vice-presidents. Above that level, the president can approve contributions up to $10 million; however, direction from the Minister is sought for all contributions between $1 million and $10 million.

IBDA projects are reviewed and recommended for approval by the IBDA management committee by consensus. Formal BDP funding approval of IBDA projects follows ACOA’s delegated authorities for the BDP, with most done at the IBD manager or the DG of IBD level, depending on the funding amount.

2.4 Expenditure Profile

The Agency’s IBD sub-program activities are funded via the BDP. The expected outcomes of the programming align with the objectives of the BDP, which are to:

Prior to April 2010, IBDA projects were administered through the International Business Development Program (IBDP). The IBDP funding was integrated into the BDP for operational efficiency purposes and to bolster an integrative approach in the delivery of the programming following the amalgamation of the Trade and Investment program sub-activities into the IBD sub-program in 2009-2010.[xi]

As shown in Table 2, total IBD expenditures from 2008-2009 to 2013-2014 were $159.8 million with $125.9 million in G&Cs and $33.9 million in O&M. The decrease over the evaluation period in both the transfer payment and O&M expenditures was mostly due to cross-government spending reviews, including a strategic review in 2010.[xii] The increase in transfer payment expenditures in 2010-2011 was largely due to the injection of short-term funding from the Government of Canada’s Economic Action Plan (EAP), including one IBD project funded through the Community Adjustment Fund.

While the salaries component of O&M remained stable, there was a steady decrease in the level of general operating costs from $1.9 million in 2008-2009 to $0.6 million in 2013-2014. The IBD sub-program used O&M to support many core trade-related activities, providing the Agency with the flexibility to address clients’ needs. During the period of this evaluation, 373 contracts were issued, totalling approximately $2.1 million in O&M, to support exporters with conference costs and to help commercial and non-commercial clients with export readiness and mentoring.[xiii] The level of contracting expenditures is comparable to that reported in the 2010 evaluation though there was a shift to a higher number of smaller-value contracts. The current average size of O&M contracts ($5,600) is much lower than that reported in 2010 ($12,865).

Table 2: IBD Sub-program Expenditures ($M), 2008-2009 to 2013-2014

Fiscal Year

Transfer Payment Program

O&M

Total

BDP*

IBDP**

Salaries

General Operating***

2008-2009

22.1

1.8

3.7

1.9

29.5

2009-2010

22.8

1.1

4.6

1.2

29.7

2010-2011

26.8

0

4.6

1.5

32.9

2011-2012

17.0

0

4.8

0.8

22.6

2012-2013

17.4

0

4.4

0.7

22.5

2013-2014

16.9

0

4.9

0.6

22.4

Total

123.0

2.9

27.0

6.7

159.6

Source: ACOA financial management system (GX system).

* One Community Adjustment Fund project ($5 million) was approved for equipment for manufacturing wind turbines in 2010-2011.

** IBDA projects were funded through the IBDP until 2010-2011; since that time, they are funded through BDP.

*** Main general operating expenditures were transportation and communication, professional and special services, rentals, purchased repair and maintenance, as well as utilities, materials and supplies.

2.5 Project Profile

Project number and type

A total of 872 projects were approved through the IBD sub-program, totalling $136.2 million from 2008-2009 to 2013-2014 (Table 3). The approved funding decreased for each of the three fiscal years after 2010-2011, influenced in part by cross-government spending reviews noted previously. There was an increase in project approvals from 2009 to 2011, largely reflecting the injection of short-term funding from the Government of Canada’s EAP.[xiv]

Table 3: Approved IBD Projects and Funding ($M), 2008-2009 to 2013-2014

Fiscal Year

Transfer Payment Program

Total Approved

BDP*

BDP (IBDA)**

ACOA $M

No. of Projects

$M

No of Projects

$M

No. of Projects

 

 

2008-2009

23.2

124

1.4

15

24.5

139

2009-2010

28.9

143

0.5

6

29.3

149

2010-2011

27.4

135

1.1

17

28.1

152

2011-2012

22.2

119

1.7

25

23.6

144

2012-2013

14.6

110

2.7

41

17.1

151

2013-2014

12.4

120

1.1

17

13.5

137

Total

128.7

750

8.5

121

136.1

872

Source: ACOA project management system (QAccess); data extracted May 2014.

* One BDP project approved in 2010-2011 was an EAP project funded through the Community Adjustment Fund. Though EAP projects are not normally part of evaluations, they are included in financial tables to provide a full picture of delivered funding.

** Prior to 2011-2012, IBDA projects were funded through the IBDP.

Table 4 presents the types of IBD projects that were approved based on a coding exercise that the evaluation team conducted to better describe project types, markets and sectors. The 267 marketing-related projects received the highest level of approved spending, at $50 million. These projects typically involve promoting products or services in an international market by developing plans and materials, hiring marketing personnel, and advertising. The second highest approved spending was for 385 trade missions (outgoing or incoming), at $36.7 million. The 54 expansion and modernization of infrastructure projects ($21.6 million) typically involved the construction or renovation of production facilities or the provision of working capital to SMEs to meet export demands.

Table 4: Number of Projects by Primary Type and Funding ($M), 2008-2009 to 2013-2014

IBD Project Type

No. of Projects

% of Projects

ACOA Assistance ($M)

% of ACOA Assistance

Marketing

267

31

50

37

Trade missions

385

44

37

27

Expansion and modernization

54

6

22

16

Partnership and coordination

28

3

13

10

Training and knowledge transfer

107

12

12

9

Planning and studies

30

3

2

2

Other

1

0

0

0

Grand Total

872

100%

$136

100%

Source: ACOA project management system (QAccess); data extracted May 2014.

Client type

According to the dollar value of projects approved, almost half (45 per cent) of IBD clients were commercial, receiving $62 million in ACOA assistance over the period of the evaluation. The remaining 55 per cent were non-commercial clients, with $74 million in ACOA assistance. Non-commercial clients included industry associations (17 per cent) and provincial governments (10 per cent). Eighteen per cent of all IBD clients were classified as other non-profit organizations, a category that includes a variety of community-based organizations such as Société Nationale de l'Acadie, port and airport corporations and LearnSphere.[xv] The remaining 10 per cent of non-commercial clients represented universities and colleges, economic development organizations, business associations and native band councils, each representing less than four per cent of total IBD assistance.

Sector

As illustrated in Figure 2, the sector that received the most support from IBD programming was aerospace and defence ($28 million), followed by agri-food/seafood ($18 million), information and communications technology ($16 million), ocean technologies ($10 million), biotechnologies and life sciences ($9 million), and environmental/clean technologies/renewable energy ($9 million).[xvi] The remaining $29.7 million was provided to a number of other sectors representing less than five per cent of total IBD assistance each.[xvii]

Figure 2: Percentage of Approved IBD Funding by Sector, 2008-2009 to 2013-2014

(D)

Source: ACOA project management system (QAccess) and evaluation team coding analysis; data extracted May 2014.

Key markets

According to project coding data, 60 per cent of clients focus on the United States market. The second most important market for ACOA clients is the European Union, with 30 per cent of projects. Clients are also seeking business in emerging markets: 18 per cent of clients are targeting Asia; another 18 per cent are looking to do business with Latin America and the Caribbean; and 9 per cent are seeking to do business in the Middle East and Africa.

Rural and urban

Although the overall number and dollar value have diminished over the scope of this evaluation, the Agency has remained consistent in its level of support to clients located in urban and rural areas. Much like what was reported as part of the evaluation of TI program sub-activities in 2010, 61 per cent of approved funding was for projects situated in urban areas, whereas 39 per cent of funding was for projects in rural areas. This proportion was not consistent across all regions, with a higher percentage of funding for rural areas approved by head office (pan-Atlantic) and the New Brunswick regional office.

3.0 Findings: Relevance

Overall, the evaluation found that the IBD sub-program remains relevant. There is a strong and continued need for IBD programming as challenges to international business development continue to persist and to some extent, have increased since the 2010 TI evaluation. ACOA is aware of these challenges and has adapted its programming to address the needs of clients by focusing on emerging markets and by helping SMEs benefit from FTAs.

IBD activities are aligned with ACOA’s strategic outcome and the ACOA Act. They are consistent with the Agency’s roles and responsibilities and with broader federal priorities related to economic development – more specifically to priorities on increasing trade and attracting investment. While recruiting and retaining foreign students is a priority of the federal government, an opportunity exists to better articulate the alignment of these activities within the IBD sub-program and with ACOA’s mandate. Likewise, the attraction of FDI is clearly aligned with federal roles and priorities although ACOA’s approach and level of involvement, as well as the types of activities undertaken and supported vary across regions.

The Agency’s IBD programming complements rather than duplicates programming being delivered by other organizations. Specific characteristics that set the programming apart from that offered by others include a focus on Atlantic Canada and on SMEs, attention to economic development outcomes, and an emphasis on federal government priorities. It also provides important financing not available from other funders. IBD programming is also complementary to other sub-programming offered by ACOA, particularly PG and IC programming.

The relevance of the IBD sub-program was assessed by examining the continued need for the programming and the alignment between the programming, Government of Canada and Agency priorities, and federal roles and responsibilities.

3.1 Continued Need for the Programming

Judgment Criteria

Key Finding

The needs that the programming is expected to meet are still present to at least the same degree as they were five years ago.

Many factors affecting the IBD have remained the same since the last evaluation: declining export activity, number of exporters, FDI and relatively low SME capacity for international business in Atlantic Canada.

Some factors have contributed to an increased need for IBD programming since the previous evaluation: global economic recovery, the need for diversification of sectors and markets, new free trade agreements, demographic changes, the importance of global value chains, provincial fiscal restraints, lagging productivity, and innovation and commercialization.

Existing strategies and policy recommendations, including Canada’s Global Market Action Plan (GMAP) and provincial IBD strategies, support the need for export development and investment attraction. Clients and KIs identify an ongoing need for IBD activities, including marketing, outreach, training, mentoring, and coordination and attraction of FDI.

IBD programming responded to emerging needs.

Information on the changing IBD landscape in Atlantic Canada was collected, analyzed and considered in decision making by ACOA and by the IBDA management committee.

KIs reported that programming was adapted over the period of the evaluation to respond to changes in the IBD environment and the need for IBD services.

KIs, clients, strategies, published research, and other literature clearly recognize the benefits and opportunities related to increasing international business development for the Atlantic Canadian economy. The Atlantic Canadian market is a small one within an increasingly global marketplace; expanding levels of international business conducted by SMEs is a key step toward an improved economy. The Government of Canada has prioritized greater business in foreign markets as evidenced by its GMAP, a series of free trade agreements and other initiatives.

As with the previous evaluation of the Agency’s TI program sub-activities, there is generally a low level of export activity and a declining number of exporters in the Atlantic region. Much of this is tied to the United States’ slow recovery from the global recession and the continued reliance on that market by Atlantic Canadian firms. It is also affected by the importance of the mining and energy sectors in Atlantic Canada, both of which are highly affected by the Canadian dollar, which has fluctuated over the period.

SMEs must consider both tariff and non-tariff barriers associated with doing business in foreign markets. While many tariff barriers are being addressed by new free trade agreements in particular markets, non-tariff barriers – regulations and product standards, distance, lack of contacts and infrastructure, language and cultural differences – remain. KIs indicated that non-tariff barriers are particularly challenging for less experienced and smaller SMEs. Several KIs noted that lack of experience with international travel also holds many SMEs back from getting into international markets.

The very small size of Atlantic Canada’s SMEs – almost 80 per cent are either self-employed entrepreneurs or have less than five employees – make it difficult for the region’s businesses to take the risks associated with international business. KIs indicated that many Atlantic Canadian SMEs prefer to stay small and to focus on domestic markets or trade activities with the United States, which are longstanding deterrents to expanding markets, increasing export activity and attracting FDI. KIs and published research highlighted that Atlantic Canadian SMEs tend to have limited resources for market research and a lower ability to take risks associated with foreign markets.

KIs and the document review reveal that some factors affecting the need for IBD supports in Atlantic Canada have increased since the previous evaluation. These include greater focus on the global economy and access to global value chains. Similarly, there is a need to diversify Atlantic Canadian sectors and their markets and address lags in productivity, innovation and commercialization. The Atlantic population continues to decline and shift, with an aging population and migration from rural to urban areas as well as to western areas of Canada. This overall population decline negatively impacts the number of SMEs and exporters in Atlantic Canada. Provincial government fiscal restraint was also an important issue over the period as the Province is a key partner on IBD activities, both financially and non-financially. KIs identified the ability to find partner resources for projects as an increasingly challenging issue.

ACOA is aware of and is responding to many of the needs associated with increasing international business, particularly SME needs for financial/non-financial assistance to enter or expand business in foreign markets. Through missions – both outbound and inbound – as well as learning and partnership activities, the programming provides low-risk opportunities to increase capacity and develop skills and networks related to international business. Overall, clients are satisfied with the programming, with 95 per cent of those surveyed stating that IBD sub-program assistance met their needs to at least some extent.[xviii]

The IBD sub-program addresses the need for greater collaboration among stakeholders in light of limited resources. In particular, the IBDA is a forum for collaboration, shared planning and the maximization of limited resources on a pan-Atlantic level. There is high satisfaction among IBDA stakeholders. ACOA and the IBDA management committee are aware and are adapting programming to address the changing context by putting more attention on emerging markets and supporting SMEs’ ability to take advantage of new FTAs. For example, four market working groups were put in place during the period of the evaluation to focus on opportunities in Brazil, India, China and the European Union.

There is evidence that program staff and managers are increasingly working with other ACOA program and policy colleagues. KIs recognized the importance of other ACOA programming such as IC and PG to achieving IBD outcomes. IBD programming is also supported by the Agency’s policy function. Throughout the period of the evaluation, ACOA’s policy function, including the work of external consultants, contributed to policy research and advice related to trade and investment. The policy function contributed to an internal IBD review that identified key factors affecting the need for the programming, with the aim of developing a more strategic long-term vision for the programming.[xix]

During the period of this evaluation, the Government of Canada increased its focus on international trade in new and emerging markets. Canada renewed its 2008 Global Commerce Strategy with the GMAP in 2013. In response, the Agency has put more emphasis on supporting new FTAs, including the Canada-European Union Trade Agreement as well as agreements with South Korea and several South American countries.

ACOA also acknowledged the shifting roles of other federal departments in the Atlantic region. In particular, KIs reported that ACOA adapted to changes to the scope of activities offered by DFATD in Atlantic Canada, as a result of downsizing, over the evaluation period. In response, the Agency’s IBD staff improved direct relationships with foreign trade commissioners located in key markets throughout the world. There were also reductions that affected the IBD-related programming offered by Industry Canada and Agriculture and Agri-Food Canada over the six-year period.

Some changing needs may not be as well understood or addressed by the programming. While the Agency did make changes to its PAA to better integrate trade and investment activities as per the previous evaluation’s recommendations, the GMAP and other policy thinking, it is not clear that this has resulted in significant integration of the delivery of trade and investment activities across the Agency. In this regard, KIs emphasized that ACOA’s role as relates to attracting FDI remains unclear and inconsistent, similar to previous evaluation findings.

It is difficult to assess whether senior management monitored or addressed all significant changes to the IBD context over the evaluation period. KI data and performance measurement data analysis revealed that some important information is not monitored consistently. Performance measurement information gaps exist that impact the ability of senior management to monitor and take action on areas related to the attraction of FDI, the acquisition and commercialization of new technology, the capacity of universities and research institutions and the development of first-time exporters. These gaps will be discussed further under section 4.0 of this report.

3.2 Alignment with Government Priorities

Judgment Criteria

Key Finding

There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome. The alignment is recognized and made explicit.

IBD programming is closely aligned with the Agency’s mandate to promote economic prosperity in Atlantic Canada.

IBD activities are aligned with federal government priorities related to economic growth, international trade and investment, including new FTAs and new and emerging markets.

The strong alignment between ACOA’s IBD sub-program and the Agency’s strategic outcome is reflected in the PAA (Appendix D). The IBD is one of three sub-programs supporting the ED program.[xx] ACOA recognizes the importance of the region’s many geographic and sectorial needs and opportunities. Through the IBD and other ED programming, ACOA helps SMEs to take advantage of free trade agreements and other trade-related initiatives and to take part in international business development.

ACOA’s IBD activities are consistent with Government of Canada priorities for “strong economic growth,” one of 16 high-level expected outcomes identified in the Government of Canada’s whole-of-government framework. This expected outcome is focused on increasing economic growth and development in all regions of the country and all sectors of the economy. The programming aligns with the federal government’s focus on international trade, particularly on expanding to new and emerging markets. As discussed previously, the programming supports Canada’s GMAP, including FTAs such as the one with the European Union.

IBD activities that support recruiting and retaining foreign students are aligned with federal government roles and responsibilities as evidenced by Canada’s Strategy on International Education. While provincial governments generally have jurisdiction over education, including post-secondary education, the federal government recognizes that attracting more international students and researchers to Canada leads to better innovation and creates jobs and opportunities. Recognizing that there are ties between increasing foreign students and immigration in general, with improved economic outcomes, the level of priority these activities have within the IBD sub-program and with ACOA’s mandate, especially in light of limited resources, is not clearly articulated or understood.

Likewise, the attraction of FDI is clearly aligned with federal roles and priorities. However, many KIs expressed uncertainty of ACOA’s role related to attracting FDI; with some believing that ACOA has no role through IBD programming. The IBD’s approach to attracting FDI is not consistently understood or implemented across the Agency.

IBD programming contributes to the government’s EAP by helping to create jobs in rural communities and by supporting SMEs. It supports Advantage Canada by helping Canadian businesses compete and by supporting skills development and the innovation vital to productivity and competitiveness.

3.3 Alignment with Federal Roles and Responsibilities

Judgment Criteria

Key Finding

ACOA is mandated by law to fulfill this role. Other jurisdictions administer such programming through the federal government.

ACOA’s roles and responsibilities related to the IBD are aligned with the ACOA Act.

Needs are not met by alternative services (extent of duplication, overlap or complementarity).

IBD programming complements rather than duplicates other economic development funding targeted at SMEs.

In keeping with the findings of the previous evaluation of the Agency’s TI program sub-activities, the IBD sub-program activities align with federal roles and responsibilities. The ACOA Act gives the Agency the authority to “plan, direct, manage and implement programs and projects intended to contribute directly or indirectly to the economic prosperity of the Atlantic region.” By investing in initiatives that support SME success in international markets, IBD programming contributes to ACOA’s mandate to “increase opportunities for economic development in Atlantic Canada and, more particularly, enhance the growth of earned incomes and employment opportunities in that region.”

The Agency’s IBD activities complement the programming offered by other organizations. As outlined in Table 5, provincial governments and other federal departments have programming that supports international business development. However, ACOA’s programming offers unique aspects, including attention to economic development outcomes, focus on SMEs and alignment with federal priorities and initiatives. The Agency’s attention to Atlantic Canada allows IBD programming to respond to specific provincial or regional needs, opportunities and contextual factors while taking advantage of synergies between the provinces. IBD programming is an important mechanism for convening Atlantic partners, including through the IBDA.

The other RDAs in Canada do not provide a valid baseline for comparison with ACOA’s programming. The other RDAs do not offer significant levels of IBD programming, due in part to the fact that other regions of Canada have higher levels of exporting and attracting foreign investment than the Atlantic region.

Table 5: Summary of Organizations Delivering IBD Programming in Atlantic Canada

Organization

Programming Characteristics

Atlantic Canada-wide scope

Alignment with federal priorities

Primary mandate supports economy and SMEs

Access to financing

Other policy and programming supports*

Supports international business development **

ACOA

x

x

x

x

x

x

Provincial Governments

 

 

 

x

x

x

Industry Canada

 

x

 

 

 

x

Foreign Affairs, Trade and Development Canada

 

x

 

 

x

x

Agriculture and Agri-Food Canada

 

x

 

 

 

x

Export Development Canada

 

x

 

x

 

x

Canadian Commercial Corporation

 

x

 

 

 

x

Business Development Canada

 

x

 

x

 

x

Source: Document and literature review and KI interview data.

* Other programming supports include policy and those aimed to improve productivity and growth as well as innovation and commercialization.

** Range of IBD activities, including funding missions with foreign markets, research and market intelligence, training and mentoring, and marketing.

Overall, IBD programming is also complementary to other sub-programming offered by ACOA, particularly to PG and IC sub-programs. According to project data and KIs, SMEs often rely on programming from one or more of the ED sub-programs as a means of entering and expanding into export markets; 58 per cent of IBD clients accessed funding from either PG or IC. As discussed previously, there is increasing coordination of program delivery, especially at regional office levels where they are supported by shared management.

KIs acknowledged that further improvements to shared planning and communication between the IBD and the other ED sub-programs, particularly at ACOA’s head office level and between the head office and the four regional offices, would be beneficial to the achievement of IBD outcomes. KIs recognized that IBD programming, historically quite separate from other programming, has become more integrated and coordinated with other Agency activities over the period of the evaluation.

4.0 Findings: Performance – Effectiveness

Overall, the evaluation found that the Agency’s IBD activities are incremental to the achievement of immediate, intermediate and ultimate outcomes. The programming achieves positive immediate outcomes, including improved awareness of international business opportunities and increased capacity among SMEs as well as strengthened partnerships and alliances. To a lesser extent, the evaluation is able to show that the sub-program achieved the intermediate outcomes of an increased number of new exporters, increased foreign direct investments, and increased acquisition and commercialization of new technology.

These immediate and intermediate outcomes contribute to the ultimate IBD sub-program expected outcome of “increased economic benefits for Atlantic Canada from international markets” as well as the ED program outcome of “improved growth and competitiveness for Atlantic SMEs.” For the most part, it is difficult to compare the results of this evaluation with those of the 2010 TI evaluation. In addition to changes to the PAA and subsequent adjustments to the PMS, there were differences in the evaluation scope, design and approaches.

Some factors continue to impede the success of SMEs in international markets, such as SME capacity and delays in reaching markets. This is specifically mitigated through the Agency’s support for skills development as well as knowledge acquisition. KIs and survey results also identified internal communication and coordination, timeliness of project approvals, and financial constraints as barriers to project implementation and success.

While improvements have been made to ensure the availability and use of performance measurement information, there is a need to address specific gaps in data. These gaps include information on intermediate and long-term outcomes and the results of non-mission activities, including those related to FDI and the education sector. The fact that the Agency no longer has access to Statistics Canada data that compares IBD-assisted SMEs with non-clients is a significant barrier to tracking results of the programming.

The effectiveness of the IBD sub-program was assessed by examining: (1) incrementality; (2) evidence of achievement of expected outcomes; (3) unexpected outcomes; (4) barriers and facilitators to achieving outcomes; and (5) evidence that performance information is adequate and effective.

4.1 Incrementality

Judgment Criteria

Key Findings

The impact of the absence of the programming is similar to or greater than that observed in the previous evaluation.

ACOA’s IBD programming, including the IBDA, are incremental to the achievement of expected outcomes. Without ACOA financial assistance, many projects would not have moved forward, and there would have been negative consequences on the scope, quality and timing of those projects that would have proceeded.

ACOA provides a large percentage of total funding to IBD activities and influences the involvement of other funding partners. For every dollar ACOA provides, $0.97 is leveraged by other project partners and collaborators, up from $0.89 in the previous evaluation.

Incrementality is an indicator that is used to measure a project proponent’s ability to proceed with a project with the same scope, within the expected time frame, without ACOA assistance. In this study, it demonstrates that the outcomes being reported are attributable to some degree to the Agency’s IBD activities and are unlikely to have been achieved otherwise.

One of the factors that impacts incrementality is the level of assistance a client receives from ACOA rather than other sources. ACOA is a significant funding contributor to IBD projects, providing 66 per cent of total project costs, followed by provincial governments at 10 percent and other federal departments at 3 per cent. The second greatest source of financial assistance for IBD projects is classified as other funding supports, which includes a proponent’s own capital and amounts to 21 per cent of project costs. As a proportion of total project funding, ACOA assistance has increased since the last evaluation by 16 per cent, and provincial support rose by 6 per cent, suggesting that the support provided by proponents themselves and by other federal government departments may have diminished.[xxi]

According to both survey and KI data, many international business development initiatives would not have proceeded without ACOA support or the projects would have been negatively impacted. Only 4 per cent of clients indicated that their projects would have proceeded as planned without ACOA assistance. Others reported that without ACOA assistance, the project would have been cancelled (40 per cent), the scope of the project would have been reduced (44 per cent), or the start of the project would have been delayed (19 per cent). IBDA respondents were more likely to report that projects would have been cancelled or that additional funding programs would have been sought in the absence of ACOA support. Client survey respondents indicated a 53 per cent likelihood that funding would still have been committed regardless of ACOA’s participation in the project.

KIs spoke of how important ACOA’s financial and non-financial assistance was to the project moving forward and to the successful achievement of outcomes. They indicated that ACOA’s assistance was particularly important for trade-related activities such as trade missions and less critical to the implementation and success of investment attraction initiatives. Some KIs noted that since investment activities tend to be led by provincial governments, the impact of not having ACOA’s contributions would not be as significant.

Financial, client survey and KI data indicate that ACOA funding influences other partners to support IBD projects. As shown in Table 6, for every dollar ACOA provided clients over the period of the evaluation, $0.91 was leveraged from other partners, up from the $0.89 reported in the previous evaluation of TI program sub-activities.[xxii]

Similar to findings of the last evaluation, leveraging is particularly strong for IBDA projects ($1.90), due to high collaboration/partnerships on projects, as well for commercial BDP projects ($1.08). In addition to funding, IBD programming also leverages non-financial resources by facilitating partnerships and coordination, the development and transfer of knowledge and skills, and the production of research and policy outputs.

Table 6: Funds Leveraged by ACOA, by Program Funding, from 2008-2009 to 2013-2014

Funding Program

Number of Projects**

Total Cost ($)

Total ACOA Assistance ($)

Total Other Federal Funding ($)

Total Provincial Funding ($)

Other Funding ($)

Leverage Ratio of ACOA Funding***($)

BDP (commercial)

227

118,258,232

56,897,691

895,508

8,309,366

13,592,260

 

1.08

BDP (non-commercial)*

523

107,380,695

65,695,497

4,109,056

8,375,262

17,172,345

 

0.63

IBDA (non-commercial)

121

24,699,970

8,510,975

1,369,455

3,090,540

11,729,000

 

1.90

Grand Total

871

      250,338,897

  131,104,163

6,374,019

  19,775,168

       42,493,605

 

0.91

Source: ACOA project management system (QAccess); data extracted in May 2014.

*The 523 non-commercial projects include 300 projects coded under ATIP, an initiative that ended in 2010.

**The Community Adjustment Fund project was not included in this particular analysis.

*** The grand total for leverage ratio is not meant to equal the sum of the leverage ratio for each funding program component.

4.2 Achievement of Expected Outcomes

Based on an analysis of all lines of evidence, IBD programming is achieving expected outcomes. Not surprisingly, the largest effects are visible at the immediate outcome level and are related to increased awareness among SMEs of international business opportunities; the increased international business capacity of SMEs; and improved partnerships and collaborations related to the IBD. While there is less evidence of increased capacity of universities and research institutions and of increased awareness among international markets of Atlantic Canada’s competitive advantages, capacities and opportunities, the program theory and implementation of activities suggest that the programming does, to some extent, contribute to the achievement of these outcomes.

The programming also contributes to intermediate outcomes, particularly increased export activity, with nearly all clients (91 per cent) reporting increased revenues and more than three quarters (76 per cent) reporting expansion of existing markets. The programming contributes to an increased number of first-time exporters to new markets though quantitative evidence is limited. There are fewer activities focused on increasing FDI and on the acquisition and commercialization of new technology, so data supporting these outcomes is also limited. The programming contributes to long-term impacts of increased employment and revenues, as well as improved competitive position.

4.2.1 Immediate Outcomes

Judgment Criteria

Key Findings

There is sufficient qualitative and quantitative evidence to argue contribution to the achievement of immediate outcomes at or above target levels, or similar or greater than the previous evaluation.

As reported in the previous evaluation, IBD programming increases awareness and capacity of SMEs to take advantage of international business opportunities. During the period of the evaluation, the Agency provided $36.7 million toward 385 mission projects, all of which included promotion and awareness activities. Furthermore, the Agency provided $12.4 million to 107 projects in support of training and knowledge transfer activities. Three quarters of surveyed clients reported increased awareness of opportunities in foreign markets; over half said they increased capacity.

According to KIs, the programming plays an important role in improving and maintaining partnerships and collaborations related to international business development. Sixty-six per cent of surveyed clients reported having developed a total of 479 new partnerships and alliances; and 63 per cent said they expanded their existing networks.

Though increasing both the capacity of universities and the awareness among international markets/investors of Atlantic Canada’s assets and opportunities is relevant to the objectives of the programming, data are too limited to demonstrate the extent of their achievement. However, based on program theory and the activities implemented, it is logical to assume that the programming contributes to some extent to these impacts.

Data collected from survey participants, KIs and existing administrative and financial systems indicate that IBD programming is achieving its immediate outcomes. The most visible outcomes are increased SME awareness of international business development opportunities, increased international business capacity for SMEs and increased partnership/coordination related to international business development in Atlantic Canada. These findings are consistent with the 2010 TI evaluation[xxiii].

Increased awareness of international business opportunities

All lines of evidence support the key role that ACOA has in increasing SME awareness of international business opportunities. More than two thirds of IBD clients (70 per cent) intended to increase their awareness of trade opportunities in selected markets; almost three quarters (74 per cent) reported an actual increase in awareness. Specifically, half of all clients reported increased awareness of investment opportunities in selected markets and 80 per cent reported that their project increased access to market information and intelligence. These survey findings are supported by the results collected from exit questionnaires, which indicate that almost three quarters (72 per cent) of mission participants learned about or accessed a new market and three quarters (75 per cent) gained a better understanding of industry opportunities.

KIs agree that increasing awareness of international business opportunities among SMEs is one of the programming’s key outcomes. KIs noted that a variety of IBD programming activities contribute to increased awareness of opportunities, including outreach activities, SME mentoring, export development programming, market research, networking activities, partnerships, trade missions and the distribution of information from DFATD’s trade commissioners.

Increased international business capacity

Evidence also supports the achievement of increased international business capacity for SMEs. SMEs increase their international business skills, knowledge and networks, leading to improved export readiness, through IBD activities related to mentoring, training and access to market intelligence. Of surveyed IBD clients, 60 per cent indicated having increased their skills and knowledge related to international business. When asked what type of knowledge was gained, IBD clients reported the following: market characteristics and profiles (47 per cent); general business practices, regulations and market expectations (27 per cent); and marketing skills (16 per cent).

In addition to trade missions, other projects that focused on planning, marketing and partnerships also have capacity-building elements. More specifically, 107 IBD projects received a total of $12.4 million in ACOA funding (9 per cent of total IBD support) for training and knowledge transfer activities over the evaluation period. Beyond the IBD, other ACOA programming contributed to SME capacity building over time – perhaps most notably through skills building initiatives supported through the PG sub-program.

ACOA uses both operational and salary dollars to support SME capacity building. For example, during the evaluation period, the Nova Scotia regional office used approximately $700,000 in operational funding to support export mentoring, export readiness and trade missions. Similarly, ACOA’s office in Prince Edward Island expended around $430,000 for export help, export readiness and trade mission support. KIs from the New Brunswick regional office noted the role of LearnSphere[xxiv] in providing support to SMEs’ capacity building as well as for commercialization activities. KIs spoke of the importance of the programming being delivered by knowledgeable ACOA staff who offer expert advice and assistance during project development, implementation and follow-up.

Though an expected outcome of the programming, there is insufficient evidence to assess the extent to which the IBD increases the capacity of universities and research institutions. According to client survey results, 17 per cent of clients aimed to increase knowledge and skills relating to international business among universities and research institutions. However, survey results do not allow analysis of change in capacity specific to these organizations. There is limited performance measurement data related to this outcome as the program does not collect information related to the capacity development of universities and research institutions on an ongoing basis.

Several KIs spoke of the importance of ACOA working collaboratively with universities and research institutions. They supported the idea that working with universities facilitates the development of entrepreneurs in general as well as the capacity of existing SMEs to enter foreign markets and expand their business there.

The recently completed IC sub-program evaluation supports the importance of SMEs working with universities and research institutions to increase their capacity for international business. The evaluation found that the Agency plays a key role in enhancing the commercialization capacity and business skills of clients. Furthermore, it suggests that there are opportunities for the Agency to further facilitate the commercialization of new technologies, products, services and processes by assisting companies to expand internationally through trade missions and other activities.

Increased collaboration and partnerships

The Agency plays an important role in increasing collaboration and partnerships related to international business development in Atlantic Canada. In addition to transfer payment projects, this is achieved through the IBDA and sector and market working groups, and through communication and coordination between ACOA’s regional offices, provincial governments and industry stakeholders.

Both the document and literature review and KIs support the importance of strong collaboration among stakeholders. Key informants report that relationships between ACOA, the provinces, other federal departments and industry associations facilitate the achievement of outcomes and maximize the utilization of limited resources. The IBDA is the clearest example of how ACOA supports collaboration on a pan-Atlantic level.

Regional offices work in partnership with provinces, industry associations, universities and research facilities on a daily basis in delivering the IBD sub-program. Perhaps the most formal collaboration at a regional office level exists between ACOA’s regional office in Prince Edward Island and the leadership of Innovation PEI. Trade Team PEI is a 20-year partnership of federal-provincial agencies and departments committed to strengthening the province’s presence in key existing markets and to diversifying the province’s interests in new, emerging global markets. Trade Team PEI’s annual work plan identifies international business development projects to be undertaken with Innovation PEI. These non-commercial projects support a suite of demand- and supply-side activities and programs aimed at helping Island-based enterprises and organizations build their capacity to pursue, compete and succeed in their international business development efforts.[xxv]

Collaboration among universities and research institutions is facilitated through the IBDA’s education sector working group. This group plans missions abroad to promote the export of Atlantic Canadian education products and services as well as to attract foreign students. Collaboration between universities and SMEs is also facilitated to a certain extent through the programming. Several KIs noted the importance of SME relationships with researchers in terms of the commercialization of products and services for export markets. While the IC sub program evaluation indicated that support from ACOA and the establishment of Springboard Atlantic has contributed to strengthening university-industry collaborations, research shows that Canada continues to lag behind other Organization for Economic Co-operation and Development (OECD) countries in these collaborations.

Many IBD projects result in increased partnerships and collaboration. Project data show that the IBD sub-program supported 28 partnership and coordination projects totalling $13 million, or 10 per cent of overall IBD funding, during the period of this evaluation. Projects may include sector-specific activities as well as conferences, events and trade mission attendance.

Over half of surveyed clients (59 per cent) said an objective of their project was to increase coordination and partnerships related to international business development and two thirds (66 per cent) reported that new partnerships or alliances had been developed to at least some extent. Fifty-three (53) clients reported the development of 479 new partnerships and alliances – an average of nine per client. Furthermore, 63 per cent of clients indicated having expanded their existing networks, partnerships and alliances. Data from exit questionnaires indicates that 45 per cent of project participants established a new relationship with another Atlantic Canadian company or organization following a trade mission.

Increased awareness of Atlantic Canada’s competitive advantages, capabilities and opportunities

There is insufficient evidence to report on the impact IBD programming has on increasing awareness of Atlantic Canada’s competitive advantages, capabilities and opportunities by markets and investors. ACOA does not track this outcome, mostly due to the complexities associated with data collection. Not only is the target population located in foreign countries, but a number of other initiatives could impact levels of awareness, therefore impacting attribution.

KIs indicated that awareness of Atlantic Canada’s assets and opportunities was achieved as part of trade missions and conferences/events as well as through collaborative efforts between ACOA and DFATD’s trade commissioner services. Since ACOA supported 385 trade mission projects, which usually include an awareness/promotion and marketing component, it is logical to expect that the programming has contributed, at least in part, to the achievement of these outcomes.

Almost two thirds (63 per cent) of surveyed clients indicated that the objective of their project was to increase awareness of Atlantic Canada’s competitive advantages, capabilities and opportunities among international markets and investors. Almost three quarters of surveyed participants said they had increased awareness of Atlantic Canada’s products, services or capabilities in targeted international markets.

4.2.2 Achievement of Intermediate Outcomes

Judgment Criteria

Key Findings

Achievement of intermediate outcomes is similar/greater to that observed in the previous evaluation:

  • increased number of Atlantic Canadian SMEs exporting for the first time;
  • increased export activity by SMEs in Atlantic Canada;
  • expanded foreign investment opportunities (FDI and Canadian direct investment abroad);
  • increased acquisition and commercialization of new technology.

Similar to findings of the past evaluation of TI program sub-activities, IBD programming contributes to intermediate outcomes, particularly increased export activity.

KIs and some survey and performance measurement data suggest that the programming contributes to developing first-time exporters and first-time exporters to new markets as well as increasing FDI and the acquisition and commercialization of new technology. Particular challenges exist related to ACOA’s role in attracting FDI that have created inconsistent understanding and approaches across the Agency.

According to survey results, performance measurement data and KI interviews, ACOA contributes to the achievement of the IBD sub-program’s intermediate outcomes. Outcomes related to increased export activity and increased first-time exporters align with outcomes reported in the 2010 TI evaluation; outcomes related to the attraction of foreign investment continue to be negligible.[xxvi]

Increased export activity

Increased export activity was assessed by looking at expansion into new and existing markets as well as export sales. Three quarters of surveyed clients (76 per cent) indicated that they had expanded their existing market to at least some extent as a result of their IBD project, while over half (57 per cent) said they had reached a new market.

Of the 1,383 mission participants who completed an exit questionnaire, 20 per cent had on-site sales during trade missions: 12 per cent under $10,000 and 8 per cent from $10,000 to $999,999. However, 67 per cent of participants identified potential sales opportunities after the trade mission, with 46 per cent anticipating long-term sales within the subsequent 12 months. Performance data reported in the DPR from 2011-2012 to 2013-2014 indicate that 631 SMEs expanded their international sales, slightly lower than the target of 670 SMEs.

KIs support the contribution of the IBD sub-program activities to increased export activity. KIs acknowledge that the programming helps SMEs overcome many barriers to exporting by increasing knowledge and awareness of opportunities and providing financial and other assistance to get them into markets. They highlighted that the increased sales and expansion to new markets takes a long time to achieve and that participation in several trade missions is usually required before establishing quality contacts and achieving sales. KIs spoke of the important role that other ED sub-programs play in increasing export activity over time through improvements to productivity, innovation, partnerships or general business capacity.

Increase in first-time exporters

The programming’s contribution to an increased number of Atlantic Canadian SMEs exporting for the first time over the six-year period is more difficult to demonstrate, partially due to changes in the PMS over time. According to performance data on new exporters reported by ACOA in the DPR from 2008-2009 to 2010-2011,[xxvii] 255 SMEs became new exporters, surpassing targets.[xxviii] Data from exit questionnaires administered to mission participants show that the programming is reaching new exporters as only 14 per cent of respondents reported export sales for the year prior to the mission.

KIs spoke of the importance of developing new exporters as well as working with SMEs over time through other programming as a way of identifying those ready to export and increasing awareness of international business development opportunities. They highlighted the important role that ACOA plays in getting Atlantic SMEs into markets for the first time, especially non-US markets. Some KIs also noted that many SMEs are going to export markets at a much earlier stage than in previous periods. Working with industry associations was noted as an important way of identifying and reaching out to SMEs that are ready to export.

Increased attraction of foreign direct investment

While FDI activities are part of the IBD sub-program outcomes, ACOA’s efforts are limited and inconsistent across the Agency, making it difficult to attribute outcomes to Agency programming. As in the previous evaluation of TI program sub-activities, KIs stated that FDI activities are mostly led by provincial governments. Within ACOA, investment attraction activities are undertaken to various degrees by ACOA regional offices, with head office being the least engaged.

According to data from ACOA’s project management system, QAccess, one in five projects (20 per cent) had a foreign investment component. Of surveyed clients, only 15 per cent indicated that they obtained additional investment in their organization from other sources as a result of their IBD projects, with $5.6 million identified as foreign investments.

Similarly, of 1,383 exit questionnaires completed by mission participants, only 20 per cent answered FDI-related questions, implying that attracting foreign investment does not apply to the majority of participants. Although few clients reported FDI activities, three quarters of surveyed clients (76 per cent) said they developed 4,828 contacts, prospects and leads following a trade mission, conference or event. While the types and purpose of prospects and leads is not specified, it is reasonable to assume that at least some are related to investment opportunities.

Many internal KIs were unclear about ACOA’s role with respect to attracting FDI. Some spoke of the need for ACOA to remain neutral and not appear to attract particular business to one Atlantic province over another. Others spoke of the important role that ACOA has in attracting investments to individual Atlantic Canadian companies rather than attracting big companies to one particular province. However, most KIs acknowledged the important role ACOA plays in promoting Atlantic Canada in the conduct of trade missions as well as the benefits to having a federal presence at FDI activities. Some KIs noted that the availability of ACOA programming related to productivity and growth or innovation and commercialization can be an incentive that assists provincial governments to attract companies to the Atlantic region.

Increased acquisition and commercialization of new technology

While ACOA has separate programming focused on developing innovation and commercialization, the evaluation attempted to assess the extent that IBD programming resulted in increased acquisition and commercialization of new technology. As discussed previously, there is a relationship between greater innovation and export success.

According to ACOA’s project management system, QAccess, only 30 of 872 IBD projects were coded as having a commercialization component, representing $6.7 million in IBD support. Commercialization activities were more prominent in sectors such as information and communications technology, aquaculture/ocean technology as well as bioscience and life sciences. Generally, these projects help businesses identify technology or equipment that will enable them to be more productive and innovative and help them meet market demands and integrate into global value chains.

Survey results support the role that IBD programming plays in bringing new products/technology into global value chains. Almost one third (32 per cent) of surveyed IBD clients said they had commercialized a total of 49 new technologies, products, processes or services in a foreign market as a result of their IBD project; 12 per cent reported having realized a total of 332 licensing/transfer agreements. The outcomes of commercialization activities and the acquisition of new technology are primarily reported through the Agency’s IC sub-program. Results from the recently completed IC sub-program evaluation show that more than half (54 per cent) of IC clients reported that aggregated commercialization revenues came from exports in the previous fiscal year.

KIs stated that acquiring and commercializing new technology often needs to happen in order to achieve other IBD outcomes. Some KIs indicated that improved coordination and communication among sub-programs could facilitate better achievement of outcomes for both IBD and IC programming by taking advantage of synergies and opportunities.

4.2.3 Long-Term Outcomes

Judgment Criteria

Key Findings

There is quantitative and qualitative data to support the actual (or potential) contribution of immediate and intermediate outcomes toward the realization of long-term outcomes such as:

  • increased employment;
  • increased revenues from international markets; and
  • other economic benefits to Atlantic Canadians.

Immediate and intermediate outcomes have shown to contribute to the realization of ultimate outcomes such as increased revenues from international markets and increased employment, which benefit Atlantic Canadians.

Ultimately, IBD programming is expected to lead to increased employment, increased revenues from international markets and other economic benefits to Atlantic Canadians. Though difficult to quantify or to attribute directly to the programming, evaluation results suggest that the immediate and intermediate outcomes of the IBD sub-program contribute to the realization of these ultimate outcomes. A large number of KIs said that ACOA contributed to longer-term outcomes including increased employment and revenues from international markets.

IBD clients report that employment was created as a result of projects. Based on client survey data, 421 people were employed full-time[xxix] (an average of 3.8 per project) and 337 part-time (an average of 3.1 per project) during the completion of the projects. Of 758 newly hired staff, 45 per cent remained employed following completion of the project, and half were retained specifically for international business development purposes. Jobs related to international business development included customer contact development, marketing, market research, manufacturing of products, project management and sales.

Clients were asked to report how much their organization’s revenues increased as a result of their IBD project.[xxx] Twenty clients reported an aggregate growth in revenues of $59.6 million in 2013-2014, and 14 clients reported a combined increase of $44.3 million from the prior year.[xxxi] Of the 23 respondents who estimated the value of increased revenues, almost all (91 per cent) reported that at least some of the increase came from markets outside of Canada ($38.7 million), while 65 per cent reported increases from other parts of Canada ($24.1 million), and 35 per cent, increases from within Atlantic Canada ($41.0 million).[xxxii] Another 28 respondents reported that it was too soon to report on increased revenues; and 55 reported that their project was not intended to generate increased revenues.

The evaluation also found some evidence of overall business improvements that occurred as a result of IBD programming. Of 144 clients surveyed, 54 per cent of respondents noted an improved competitive position as a result of their project, including improved status/reputation (33 per cent), improved knowledge/capacity building (27 per cent), and expansion into new markets/increased sales (21 per cent).

4.3 Unintended Outcomes of IBD Programming

Judgment Criteria

Key Findings

No judgment criteria

Two unexpected outcomes were identified:

Work conducted through the IBDA, and more specifically through sector working groups, contributes to the growth of some sectors on a global scale.

While repeat missions are important for the development of relationships and credibility within global markets, there is a risk that some less experienced SMEs may be excluded from participating and that ACOA could be missing opportunities to support other events and activities.

KIs were the main source of information on unintended outcomes of the IBD programming. They noted that work conducted through the IBDA, and more specifically through sector working groups, contributed to the growth of Atlantic Canadian sectors on a global scale. The two notable sectors that are recognized as more international are ocean technology and the life sciences sector.

The review of project data and KIs confirmed that there are many instances of repeat missions to the same event every year. The repeated attendance at trade mission events provides SMEs the opportunity to develop their presence and solidify their credibility in international markets. However, several KIs noted that repeat missions may prevent new SMEs from participating in trade missions and the exploration of different missions or events. Some KIs suggested that it is important to balance repeat missions with those that allow exploration of new markets and opportunities.

4.4 Barriers and Facilitators to the Achievement of IBD Outcomes

Judgment Criteria

Key Findings

Evidence that factors impeding the success of the IBD sub-program are known and mitigation strategies are implemented where appropriate.

Evidence of factors facilitating the achievement of IBD expected outcomes, and the application and sharing of best practices where appropriate.

Consistent with the 2010 TI evaluation, the most common barriers to the success of projects were found to be SME capacity to conduct international business, SME culture in Atlantic Canada that prefers to remain relatively small and focus on domestic markets, technical and regulatory barriers, and local and global market conditions. Timing and financial constraints also remain key barriers.

In addition to financial support, ACOA’s non-financial assistance and input from other ED sub-programs were facilitators to the achievement of IBD expected outcomes. Other facilitators are strong SME leadership and capable staff, sector-specific networks, partnerships and contacts.

Barriers

According to surveyed clients and KIs, barriers to the achievement of IBD outcomes have remained similar to those mentioned in the 2010 TI evaluation. These barriers include timing and financial constraints, SME capacity and cultural challenges, international regulations and the state of the local and global economy.

Over half (51 per cent) of surveyed clients did not respond to questions related to barriers they experienced in the achievement of expected outcomes. More than three quarters (78 per cent) of the 58 clients who did respond indicated the following specific factors:

KIs corroborate these findings and noted other important factors that impact export and investment attraction. For example, KIs spoke of technical and regulatory challenges related to emerging markets. They also indicated that the small size of SMEs affects their capacity to participate in missions or take risks associated with international business. KIs noted that many SMEs lack the time or resources to attend all missions abroad, and those that do participate on a recurring basis can experience mission fatigue – which can mean they are not able to be as strategic or may not have the time and resources to adequately prepare for or follow-up on missions. Several KIs referred to a culture in Atlantic Canada in which SMEs tend to be satisfied as a small business with few employees and to focus on the local or domestic market. The level of comfort that SME owners have with international travel, particularly related to language diversity and culture, was also noted as a barrier that is often mitigated by IBD programming.

Facilitators

KIs and clients identified factors that facilitate the achievement of positive outcomes, including those related to the delivery of IBD programming and to the capacity and networks of SMEs. KIs highlighted that delivery factors such as decentralized programming, financial support and non-financial IBD program assistance facilitate the achievement of results. The IBD delivery model allows capacity development, from project development to longer-term follow-up. Account managers are highly involved with IBD projects, have strong relationships with SMEs, and offer meaningful advice and expertise to guide clients through their business development.

Surveyed clients also noted the importance of non-financial supports provided by ACOA. Sixty-two per cent of clients said that non-financial assistance such as strong knowledge and guidance from ACOA staff, as well as training and mentoring activities helped them achieve their project objective. Fifty-eight per cent of clients specifically noted the value of assistance provided for trade missions, including before, during and after such activities.

The delivery of IBD programming through regional offices promotes collaboration with other Agency sub-programs. Both clients and KIs acknowledged that inputs from IC and PG programming assist in the achievement of IBD expected outcomes. As already reported as part of incrementality, some KIs stated that many IBD projects would not occur without inputs from other programming. In addition to the other ED sub-programs, ACOA’s CD and PAC programming also contribute to the expected outcomes.

As shown in Figure 3, 58 per cent of IBD clients accessed funding from the IC or PG sub-programs over the evaluation period. While only one IBD client accessed PAC sub-program support for supply chain and market research through G&Cs, key informants noted that PAC often provides supports through O&M to IBD staff and managers, which, in turn, benefits clients and other stakeholders.

Figure 3: IBD Clients Accessing Other ACOA Funding Programs, 2008-2009 to 2013-2014

(D)

Source: ACOA project management system (QAccess); data extracted May 2014.

Similarly, client survey participants were asked if they had made use of any other ACOA-supported resources when implementing their project and, if so, how they rated[xxxiv] those resources in terms of level of contribution to project results. Half of all IBD clients (50 per cent) said they received assistance related to productivity and growth and that this was a factor in the success of their project. The figure was 38 per cent for those who mentioned supports related to innovation and commercialization. Clients highly rated the supports from Productivity and Growth as well as those from Innovation and Commercialization sub-programs (4.1) - on a 5-point scale.

Key informants spoke highly of strong relationships among stakeholders as being a key facilitator to achieving positive IBD outcomes. They identified the IBDA as a best practice, including the various market and industry sector working groups. However, several KIs noted that some of the sector working groups are stronger than others and that those that have more industry leadership and a business planning approach tended to be more organized and effective.

Clients were also asked about key success factors to achieving IBD outcomes. The most frequent facilitators that they reported were having strong leadership, capable staff, management or consultants (34 per cent) and the availability of networks, partners and contacts within the sector (26 per cent). Other facilitators reported by clients were financial support from others (18 per cent), non-financial support from others (16 per cent), and offering a good product or service (10 per cent).

Best practices and lessons learned

Best practices that have proven track records for achieving positive outcomes were considered through analysis of KI and other data. A key best practice of the IBD sub-program is its collaboration with external stakeholders. In particular, the broad collaboration among stakeholders that is achieved through the IBDA is an important element of the programming. This collaboration, which includes ACOA and other federal government departments, the four Atlantic Provinces and industry stakeholders, is an important mechanism for maximizing limited resources. The sector and market working groups were also highly praised for prioritizing key sectors in Atlantic Canada and allowing for shared planning and implementation of initiatives.

The collaboration that occurs through regional ACOA offices as part of the delivery of IBD activities is also strong and important. Regional ACOA staff and management work closely with provincial partners, with industry associations and with universities and research institutions to plan and implement activities that will lead to improved expected outcomes. Of all the regional offices, ACOA’s regional office in Prince Edward Island may have the highest level of collaboration with the provincial government given its unique delivery through Trade Team PEI.

The evaluation notes several promising practices – practices that show a great deal of potential but that may be in early implementation stages or not yet broadly implemented throughout the Agency. These are important practices to highlight for application more broadly across other regional offices or programs, or at an Agency-wide level. One promising practice highlighted by some KIs was considering more inbound missions. KIs from the Prince Edward Island regional office spoke about the success experienced by the bioscience/life sciences sector in hosting incoming trade missions. They noted that the incoming missions, which generally consist of a conference with matchmaking meetings and other events, provide unique opportunities to promote the Atlantic region. They allow clients to demonstrate their capacity and products in ways that cannot be done by attending an overseas trade mission.

Some practices demonstrated by the Newfoundland and Labrador and Nova Scotia regional offices also hold promise. Newfoundland and Labrador regional office stakeholders spoke of building a strong internal organizational culture of program integration. Office staff there work on initiatives through TeamWorks groups, which are comprised of members from different program teams. This regional office has also piloted an approach to develop SME awareness of opportunities and capacity for the IBD through the network of Community-based Business Development Corporations located throughout the province.

KIs from the Nova Scotia regional office indicated that the attraction of FDI was a key role of both the IBD and other ED staff. They provided examples of promoting the potential for partnerships as well as for investments in existing companies to external investors and firms, and for setting up businesses to take advantage of regional capacities. The Nova Scotia office also initiated support for an annual venture-capital forum during the evaluation period that shows promise in assisting SMEs with the attraction of foreign investment.

A lesson learned is knowledge or understanding gained by an experience, which may be positive or negative, and can be applied to future situations. A key lesson identified by KIs is that project failure is an opportunity for a client to reassess and readjust needs and strategies to better meet its objectives. This could mean focusing on a different market that holds more opportunities. By proactively mitigating future problems and redirecting clients on their path to being productive, innovative and competitive, not achieving expected outcomes can ultimately lead to overall success.

4.5 Adequacy of Performance Measurement

Judgment Criteria

Key Findings

IBD sub-program performance measurement is adequate and effective in reporting on the achievement of outcomes.

Performance measurement information is considered useful and is being used by the Agency and by ACOA-funded organizations.

Performance information is collected and is available for use during the evaluation.

Since the previous evaluation, progress has been made in ensuring the availability and use of performance measurement information.

The program uses data, particularly from the exit and one-year-out questionnaires completed by mission participants, to make decisions about the program.

There are gaps in performance data, however. For example, data related to non-mission activities is limited as is data that can provide early indicators of success. There are also challenges with the key performance data collection tools and methods.

Important progress has been made since the previous evaluation of ACOA’s TI program sub-activities to ensure that performance measurement information is available to support evidence-based decision making, the communication of results and the completion of evaluation studies. The program implemented standardized exit and one-year-out questionnaires during this period of evaluation, allowing for better collection of data from mission participants.

KIs recognize the importance of collecting and using performance measurement data and the need for standardized approaches to data collection and reporting. However, they identify several challenges with collecting performance data, including the length of time needed to achieve IBD expected outcomes and the influences of other initiatives. Indeed, the development of capacity, awareness, the identification of prospects and leads, building partnerships/alliances and sales, all take time to achieve, and the current performance measurement approach lacks contextual information to show progress on the path to the achievement of outcomes.

Other concerns related to performance measurement are the amount of data collected, the limited breadth of data in relation to all IBD activities and the use of paper-based questionnaires. Key informants said that the current PMS focuses too much on export outcomes and fails to capture important data on the development of partnerships, alliances and agreements, as well as technology acquisition and commercialization. Additionally, there is a lack of standardized project indicators enabling the roll-up of common indicators into intermediate and longer-term outcomes. This issue, however, is not specific to the IBD sub-program, but rather crosses all sub-programs.

An analysis of project data collection tools also revealed that the results management tracking system has not been used consistently across regions. While it holds promises to capture valuable outcomes data, the majority of the fields that have been populated are limited to output-level information. Additionally, the current evaluation discovered that data entry related to some non-commercial projects monitored in the project (QAccess) system are still coded to the Atlantic Trade and Investment Partnership, which no longer exists.

As previously discussed in Section 1.3, availability of performance data was a challenge for this evaluation. An assessment of existing data showed limited performance outcome information on the following indicators: increased number of Atlantic Canadian SMEs exporting for the first time; increased international business capacity for university/research institutes; attraction and retention of foreign students; increased awareness among international markets and investors of Atlantic Canada’s competitive advantages, capabilities and opportunities; and increased acquisition and commercialization of new technology. Without the ability to track these activities and results on an ongoing basis, senior management’s understanding of program outputs and outcomes is constrained.

As previously discussed in section 2.3 of this report, changes in the methodology used by Statistics Canada prevents comparison of export sales between assisted and non-assisted firms. This problem hampered the Agency’s ability to report on program results beginning in the 2013-2014 DPR. It also negatively affected the ability of this evaluation to assess ACOA’s contribution to the achievement of outcomes.

5.0 Findings: Performance – Efficiency and Economy

Overall, the evaluation found that the IBD programming demonstrates efficiency and economy in the utilization of resources.[xxxv] The cost of delivering IBD programming has remained relatively stable since the previous evaluation. While the cost of delivering the activities is higher than for other ACOA programming, the evaluation recognizes that this is related to aspects of the delivery model that are important to the achievement of expected outcomes.

Mechanisms are in place to ensure the efficient use of resources, including governance structures, planning and budgeting processes, and a decentralized delivery model. One area where efficiency and economy could be enhanced is through better articulation of Agency priorities, activities and expected outcomes of IBD programming as some weaknesses related to planning, internal communication and coordination pose risks to the strategic achievement of outcomes.

The evaluation assessed efficiency and economy by examining the following: (1) evidence of efficient utilization of resources, (2) the existence of mechanisms that promote efficiency, including those related to governance, planning and budgeting, and delivery, and (3) the extent to which the program considers alternative delivery mechanisms, lessons learned and best practices.

5.1 Efficient Utilization of Resources

Judgment Criteria

Key Findings

Delivery costs across regional offices have remained stable or decreased since the last evaluation and compare reasonably to one another within the context of factors affecting regional office delivery.

Overall, the IBD sub-program delivery costs have decreased over time. Compared to other ACOA programming, there is a relatively low number of staff involved with its delivery; staffing expenditures have remained steady over time.

O&M costs have declined over time, in part due to decreased travel and face-to-face meetings but also because of changes in the way some regional offices support SME capacity-development activities.

ACOA’s cost in delivering $1 in G&Cs is reasonable compared to the previous evaluation and to the delivery of other programming.

Delivery costs are reasonable compared to the previous evaluation. ACOA’s cost to deliver $1 G&Cs under the IBD was $0.27, slightly less than the cost calculated for the period of the previous evaluation ($0.29).

IBD activities are more costly to deliver compared to other ACOA programming due to a high level of staff involvement in the development, implementation and follow-up of projects; international travel costs; and the use of O&M to develop SME capacity for international business development.

Delivery costs are comparable to the 2010 TI evaluation. ACOA’s cost of delivering $1 of G&Cs under the IBD was $0.27, slightly less than the cost calculated for the period of the previous evaluation ($0.29). Overall, the IBD sub-program delivery costs have decreased over the six-year evaluation period.

While IBD programming represents only 15 per cent of ACOA’s overall expenditures, with relatively limited G&Cs and number of staff involved with program delivery, several factors make it more costly to deliver compared to other ACOA programming. The cost of delivering $1 of G&Cs for IC sub-program activities from 2005-2006 to 2012-2013 was approximately $0.09 in O&M. The higher IBD programming costs are due to specific factors associated with its delivery: the high level of IBD staff involvement in the development, implementation and follow-up of projects; international travel costs; and the use of O&M for SME capacity development initiatives.

As previously shown in Table 2, section 2.4, both G&Cs and O&M expenditures decreased over the six-year period, mostly due to cross-government spending reductions, including a strategic review in 2010.[xxxvi] The Agency-wide salary component of O&M expenditures remained stable over the period though the Nova Scotia region had significant drops in salary in 2012-2013 and 2013-2014, in part related to an unfilled account manager position, while head office and the Newfoundland and Labrador regional office showed steady increases each year. Though the number of staff involved in IBD programming has decreased, salary expenditures have remained steady, reflecting annual pay increases according to collective agreements.

General operating expenditures, particularly those related to transportation and professional and special services, decreased across all regions and head office, in part due to government-wide expenditure reduction exercises. Another factor that influenced the decrease in general operating expenditures was the change in how SME capacity development activities are supported in some regions. For example, the Nova Scotia and Prince Edward Island regional offices were partners on larger-scale O&M projects to deliver skills development training activities in past years. They now concentrate more on smaller, SME-specific capacity building initiatives.

5.2 Structures and Mechanism that Support Efficiency and Economy

Judgment Criteria

Key Findings

ACOA has in place resource optimization mechanisms to ensure that the most efficient and economical means are used to administer the programming.

Findings are similar to the previous evaluation with respect to the importance of a decentralized delivery model as well as the need for greater internal communication and coordination.

There is an opportunity to enhance efficiency and economy of the IBD programming by clearly articulating the Agency’s priorities, main activities and expected outcomes as well as by making further improvements to planning, internal communication and coordination.

Multiple mechanisms related to governance, delivery, planning and budgeting currently exist to support the efficient and economical delivery of the IBD sub-program.

Governance and delivery mechanisms

In general, the efficient and economic delivery of programming is facilitated by clear governance, including well-defined roles, decision making and procedures. As outlined in section 2.3, ACOA operates within a decentralized governance structure and IBD activities are planned and implemented by both the head office and the four regional offices. Led by the DG of IBD, ACOA’s head office administers IBD funding projects that are pan-Atlantic in nature, both through the IBDA as well as through a separate allocation of BDP funding. Head office is responsible for supporting the IBDA management committee, with a lead manager assigned to the IBDA, and by managing the secretariat function. It also plays an important leadership and coordination role in terms of planning and performance measurement.

Regional IBD teams work with their regional ACOA colleagues and provincial international business development stakeholders on program planning and delivery. IBD staff in regional offices are uniquely positioned as they have long-term relationships with SMEs and understand their province’s unique assets and needs.

In keeping with these relationships, the regional offices are involved in the IBDA management committee and in the sector and market working groups even though head office staff have lead ACOA roles. Regional offices share a vote on funding decisions as part of the IBDA management committee, with the vote rotated between the four ACOA offices at each meeting.

Key informants reported that the IBD’s governance structure has important strengths in terms of allowing regional ACOA staff to develop strong knowledge of provincial stakeholders, particularly the SMEs, as well as assets, needs and other realities. However, KI interview data show that planning, communication and coordination could be strengthened between ACOA offices and sub-programs to ensure the maximization of resources and sharing of best practices and lessons learned.

Strong coordination of ACOA’s various areas of programming is an important aspect of delivery. Both KI and project data show that IBD outcomes often require input from a variety of programming areas across CD, ED and PAC program activities. While ACOA adopted an integrated approach to the IBD in 2010 by amalgamating the TI program sub-activities, evidence suggests that coordination and sharing of information remains somewhat fragmented between the IBD and other sub-programs and between the head office and regional offices.

Coordination and communication across sub-programs appears to be stronger at a regional level than at ACOA’s head office. Given the existence of both a DG of ED and a DG of IBD, KIs recognized the importance of using existing governance and reporting mechanisms to foster greater collaboration in planning and coordination at the head office level.

Some KIs indicated that existing pan-Agency management committees, including the IBD working group, ED directors committee, and DG Operations committee, could be more effectively used to plan, communicate and coordinate IBD activities across the Agency. The relatively newly formed Canada-European Union Trade Agreement working group was named as a promising practice for cross-Agency planning and collaboration.

Most KIs cited the IBDA management committee and the 12 sector and four market working groups as important mechanisms for coordination and communication among government, industry and research communities. Enhanced collaboration is especially important for the maximization of limited resources among various stakeholders.

Recognizing the various levels of development and strengths related to the governance of the IBDA working groups, some KIs stated that there is an opportunity for better sharing and implementation of best practices related to how the working groups function. Particular suggestions were to ensure that working groups have strong industry leadership, including a manageable number of members, and develop and implement longer-term strategic and business plans. In light of changes to some government departments and how management committee processes have evolved over time, many KIs also highlighted the opportunity to review the membership and other governance practices as part of the 2016 IBDA renewal process.

Some KIs expressed concerns related to the lack of obvious differences in the scope and processes associated with the IBDA versus pan-Atlantic BDP projects. Many KIs acknowledged that IBDA planning and budgeting processes were strengthened in 2013-2014 in response to the IBDA running out of funding in previous years, which then led to the use of head office BDP funds for projects that may have normally been funded through the IBDA. It is not clear whether or not any spillover between these two types of pan-Atlantic BDP funding poses specific risks. However, it does create potential for confusion among program delivery staff and clients and could lead to inefficiencies in terms of duplication of effort with two approval processes related to similar types of projects.

Planning and budgeting mechanisms

Strong and coordinated planning, budgeting and reporting mechanisms support the efficiency and economy of programming. Key informants noted that better Agency-level IBD planning and priority setting could enhance efficiency and economy of programming. They also highlighted the importance of recognizing regional differences in terms of relevant priority sectors and other factors in the development of IBD strategies or plans.

Key informants noted several improvements to IBDA planning and budgeting over the period of the evaluation. To address budget shortfalls in previous years, the committee moved to a fall planning process for all working groups, which enabled their top five projects to be presented to the IBDA management committee during the winter in 2013-2014. A weakness related to IBDA planning and budgeting is the limited ability to respond to mid-year opportunities, such as new events. The absence of an IBDA-level strategic plan means that there is no integrated approach in terms of targeted sectors and markets. The IBDA rather relies on projects submitted by the 16 working groups.

There is evidence that IBD management uses performance information to assist in planning and project decision making, including by the IBDA management committee. Performance data is also used to support Agency reporting requirements. As discussed previously, some specific gaps in performance measurement information weaken its use for planning, budgeting and reporting.

5.3 Alternative Modes of Delivery

Judgment Criteria

Key Findings

Evidence that program management has considered and continues to explore alternative modes of delivery within and outside the organization.

ACOA’s program management has considered and continues to explore alternative modes of delivery such as:

  • merging separate TI program sub-activities into one IBD sub-program;
  • increasing focus on new and emerging markets; and
  • initiating an IBD review to determine the future approach for the programming.

The IBDA management committee has implemented new processes to identify IBDA projects and has established new market working groups since the last evaluation.

Key informants believe that the delivery of IBD programming is appropriate and a cost-effective way to support SMEs to do more business in international markets. The delivery of the IBD programming through head office, regional offices and the IBDA allows the Agency to respond to regional opportunities and challenges, collaborate with partners, and develop deep knowledge and relationships with SMEs.

Key informants indicated that ACOA program management is aware of shifting contextual factors and has considered implications on programming. As a result of the findings outlined in the previous 2010 evaluation of TI program sub-activities, ACOA amalgamated the TI program sub-activities into the IBD sub-program, reflecting a policy shift toward an integrated trade approach. Some KIs, however, questioned the extent to which the delivery of the programming has actually changed and called for more clarity in relation to the Agency’s role in investment attraction.

ACOA increased its focus on new and emerging markets and supported the implementation of new federal trade agreements as per federal strategies and policy advice over the evaluation period. For example, ACOA program management has implemented an internal working group to better understand the implications and opportunities related to the Canada-European Union Trade Agreement. The IBDA management committee has added four market working groups focused on Brazil, India, China and the European Union to strategically target programming on opportunities in these markets.

As noted under the relevance section of this report, ACOA has been proactive in assessing the factors affecting international business development in Atlantic Canada as a way to set future direction for the sub-program through an internal program review. The IBD review, initiated in 2012-2013, examined the economic context, federal policies and other contextual factors.At the time of this evaluation, the IBD review was on hold in order to consider the further information and analysis that this study would provide related to program governance, the attraction of FDI and the development of new exporters, and program results.

The IBDA management committee has made a number of changes to improve project planning and approval processes. These include implementing a revised project development and review procedure in which each working group prioritizes five project proposals for the management committee to review once a year. While overall feedback was positive, some KIs said that the new process weakens the Agency’s ability to address opportunities that arise at other times of the year and does not prioritize the activities of any of the sector or market working groups over the others.

Beyond the selection and provision of support to the 16 sector and market working groups, the IBDA does not articulate cross-cutting priorities or strategies. On a related note, the high number of sector and market working groups poses risks to the efficient use of resources. Several KIs noted that some working groups, particularly ocean sciences and bio-life, are stronger than others. Implementing standardized management practices and sharing best practices would be useful. Some KIs suggested that the IBDA working groups, which involve a variety of government, industry and research stakeholders, could be better utilized to support collaboration on the achievement of Agency outcomes beyond the IBD.

6.0 Conclusions and Recommendations

The IBD evaluation conclusions and recommendations were identified through analysis of the key findings presented in this report. They were validated through discussion and consultation with the Evaluation Advisory Committee, ACOA senior management and other stakeholders.

6.1 Conclusions

Overall, the evaluation finds that ACOA’s IBD sub-program continues to be relevant and effective. There is a demonstrable need for the export and investment activities and they are aligned with the Agency’s mandate and Government of Canada’s roles and priorities. Changes in the IBD environment include the Government of Canada’s increased focus on international business development as well as on the efficient use of resources. ACOA is aware of changing needs and has adapted the programming accordingly. IBD activities complement rather than overlap or duplicate other internal and external programming.

The sub-program is successful in achieving expected results, especially at the immediate level in keeping with program theory. Programming leads to increased SME awareness of international business development opportunities, SME capacity, and improved partnership and coordination. While IBD programming contributes to expected intermediate outcomes, there are questions related to the extent to which it leads to an increased number of first-time exporters, greater attraction of FDI and increased acquisition and commercialization of new technology.

IBD outcomes are facilitated by the program’s non-financial elements and other ACOA supports that improve the capacity of SMEs for international business over time. A key best practice is collaboration with various partners, including through the IBDA. Further efforts to plan, coordinate and integrate program strategy and delivery holds potential to maximize a client-centred approach and the achievement of outcomes.

Performance measurement has improved over the period of the evaluation. Further adjustments are needed to ensure management has timely access to key data to support decision making into the future. Key limitations related to performance measurement are the availability of comparable export sales data from assisted and non-assisted SMEs, the narrow range of performance indicators, and cumbersome data collection mechanisms.

The cost of delivering IBD programming has remained stable since the previous evaluation, though it is higher than for other ACOA programming due to the greater involvement of ACOA staff and the use of O&M to support project planning, implementation and follow-up. These supports are seen as important facilitators of SME capacity for international business and for the achievement of other expected results.

While governance, planning and delivery mechanisms are in place to support the efficient utilization of resources, there is an opportunity to mitigate a number of risks that could interfere with ACOA’s ability to fully achieve IBD outcomes in the future. In particular, the lack of an articulated IBD strategy that sets out ACOA’s roles and priorities poses a risk to the achievement of outcomes in the future. Related to this risk are opportunities to improve internal communication and coordination. In articulating IBD priorities and approaches, it will be important to maintain an appropriate balance between being strategic and focused across the Agency while allowing for innovation and regional flexibility.

6.2 Recommendations

The three evaluation recommendations have been discussed with the members of the Evaluation Advisory Committee and senior management. Each recommendation aims to build upon progress made to program delivery, performance and efficiency since the previous evaluation, while ensuring that ACOA identifies and considers emerging programming needs on an ongoing basis. Table 7 presents the linkages among findings, conclusions and recommendations.

Recommendation 1: ACOA should build on its corporate knowledge and current best practices in the IBD to develop and communicate a corporate IBD strategy. A clearly articulated and communicated strategy for IBD programming should:

Recommendation 2: To ensure that the governance framework supports evolving interests and needs, including those that emerge from the development of an IBD strategy (Recommendation 1), ACOA should review the governance structures for IBD programming.

Strong governance structures promote clear decision making, communication and coordination of programming within the Agency. The governance review should assess roles and responsibilities related to IBD programming. It should identify mechanisms to ensure communication and shared planning between the IBD and other ACOA programming, within ED as well as CD and PAC.

In light of the upcoming renewal of the IBDA, and building upon existing strengths and mechanisms, ACOA should also take a leadership role in the review of the IBDA governance to ensure appropriate representation on the management committee and that processes related to planning and decision making are in place.

Recommendation 3: To support results-based management, ACOA should build on its previous efforts to improve the IBD performance measurement strategy by reviewing and updating the program’s indicators and expected outcomes, and revising tools and processes.

The program has made important improvements to performance measurement since the previous evaluation including the development of standardized participant questionnaires. These existing tools and processes can be built upon to streamline data collection and reporting of key performance information needed for decision making and accountability. IBD management should continue with efforts to further streamline and develop an electronic format for the IBD participant questionnaires. Furthermore, there is an opportunity with the current review of all ED sub-program performance measurement strategies to better align such strategies across the Agency by identifying common expected outcomes and developing standardized indicators and data collection tools.

Table 7: Alignment of IBD Evaluation Findings, Conclusions and Recommendations

Summary of Key Findings

Summary of Conclusions

Summary of Recommendations

Relevance

  1. IBD activities are relevant to the needs of Atlantic Canada SMEs and other stakeholders. ACOA is aware of changing needs and has adapted programming accordingly.

  2. IBD programming is aligned with federal roles and priorities. Overall, the programming aligns with the Agency’s mandate and the ACOA Act; clarification of IBD priorities, roles and expected outcomes is needed regarding FDI and the attraction of foreign students.

  3. While IBD activities complement rather than overlap or duplicate other internal and external programming, opportunities exist for Agency strategy setting and greater internal coordination.

  4. The Agency’s IBD programming is incremental to the achievement of expected outcomes; its absence would have negative consequences on the scope, quality and timing of projects.

  5. Programming leads to increased SME awareness of international business development opportunities, SME capacity and improved partnership and coordination. While IBD programming contributes to expected intermediate outcomes, there are questions related to increasing first-time exporters, attraction of FDI, and the acquisition and commercialization of new technology.

  6. IBD management is aware of factors that impede the achievement of outcomes. Further efforts to improve internal communication and to integrate program delivery holds potential for maximizing the achievement of outcomes.

  7. IBD outcomes are facilitated by the program’s non-financial elements and other ACOA supports that improve the capacity of SMEs over time. They are also facilitated by strong collaboration with various partners, including the IBDA.

  8. Improvements have been made to performance measurement since the previous evaluation; further efforts are needed to address key challenges to the availability of information for senior management decision making.

  9. While the delivery cost of the IBD is higher than for other ACOA programming, the high level of ACOA involvement in planning and delivery is a key success factor. Costs have remained stable since the previous evaluation.

  10. Mechanisms exist to support the efficient delivery of the programming. Further improvements could be realized by identifying and communicating clearer IBD program objectives and priorities across the Agency while strengthening linkages across programming.

  1. Develop and communicate an IBD strategy within the Agency to promote a better understanding of program objectives and priorities within the Agency and beyond. (Linked to Conclusions 2, 3, 5, 6, 7, 8, 9, 10)

  2. Review and clarify the current governance structures for IBD programming to ensure they promote clear decision making, communication and coordination of programming within the Agency. (Linked to Conclusions 3, 5, 6, 7, 8, 10)

  3. Further improve the IBD performance measurement strategy by reviewing and updating the program’s expected outcomes and indicators and by revising tools and processes to improve the availability of data that will support senior management decision making. (Linked to Conclusions 2, 5, 8, 10)

 

IBD activities address the key GC priority on expanding trade, including FTAs and emerging markets, by providing direct supports to SMEs and facilitating collaboration among stakeholders. ACOA’s roles related to the capacity of universities and the attraction of foreign students are not clearly articulated.

SMEs continue to need support for international business development. Most factors that challenge the ability of SMEs to export and attract investment remain or have increased since the previous evaluation.

ACOA has adapted programming to address changing needs: merging TI sub-activities; increasing focus on emerging markets; and supporting FTAs.

IBD programming offers unique qualities: Atlantic Canada scope and regional office delivery; knowledge of SMEs; and financial and other supports. There are opportunities to improve internal coordination and communication.

Effectiveness

IBD programming is important to the implementation of activities and achievement of results. Only four per cent of projects would have proceeded as planned without ACOA assistance. For every ACOA dollar invested, $0.97 is leveraged from other funding partners.

The programming achieves key immediate outcomes: improved SME awareness of international business development opportunities; increased capacity of SMEs; and improved partnership and collaborations. There is less evidence related to increased capacity of universities and awareness by international markets/investors of Atlantic Canada’s assets/opportunities.

The programming contributes to intermediate outcomes, with 76 per cent of clients reporting expansion of existing markets; 91 per cent, increased revenues. While the programming contributes to an increased number of first-time exporters and first-time exporters to new markets, as well as to increasing FDI and the acquisition and commercialization of new technology, data is limited. The programming contributes to longer-term expected economic impacts.

ACOA is aware of barriers to the achievement of outcomes; IBD programming mitigates the key barriers of SME capacity, access to financial supports and difficulties related to emerging markets. Facilitators include financial and non-financial assistance (before, during and after trade missions); PG & IC inputs; and strong, organized working groups.

Best practices are collaborations, including through the IBDA and sector/market working groups; regional delivery; and a client-centred approach. Promising practices include smaller, focused missions; incoming missions; and integrated planning/delivery of some regional programming.

Changes to client questionnaires have improved the availability of performance measurement data. Issues remain: narrow range of indicators; long, complicated questionnaires; difficulties related to attribution, including the time frame to reach outcomes; issues with Statistics Canada’s comparison model; and a lack of common indicators across ACOA programming.

Efficiency and Economy

Overall, delivery costs decreased over time. ACOA’s cost of delivering $1 in G&Cs under the IBD is $0.27, which is higher than the cost for other programming. Factors that increase the cost include high staff involvement, international travel, and the use of O&M for SME capacity supports.

Several aspects of the programming promote efficiency: decentralized delivery; collaboration among external partners, including shared planning and delivery between ACOA and the provinces; and strong sector and market working groups.

The lack of corporate strategy, shared planning, communication, internal coordination between head office and regional offices and the absence of strong collaboration between the IBD and ED at head office negatively affects program efficiency. IBDA planning has improved, though an overarching strategy beyond sector and market working groups does not exist.

Appendix A – IBD Evaluation Management Action Plan

Recommendations

Management Responses

Planned Actions

Responsibility

Target Date

1.  ACOA should build on its corporate knowledge and current best practices in IBD to develop and communicate a corporate IBD strategy. A clearly articulated and communicated strategy for IBD programming should:

  1. provide corporate strategic direction while allowing for regional variability and flexibility;

  2. outline the Agency’s strategic direction and approach to IBD programming, including the types of projects and initiatives that will be supported, program stakeholders, including clients, collaborators and beneficiaries, and expected outcomes. A strategy should articulate any differences between IBD programming delivered through ACOA’s head office, regional offices and the IBDA as well between other Agency programming; and

  3. promote a strategic approach to coordination across offices and sub-programs in an effort to achieve and report IBD outcomes that cut across ACOA’s programming and priorities. The strategy should identify and promote best practices, and improve coordination and communication in support of achievement of expected outcomes.

Agree

ACOA will develop an International Business Development Strategy, building on corporate knowledge, current best practices and federal priorities. This strategy will be developed through a coordinated approach between head office and the regional offices, with stakeholder engagement. A risk assessment will be included as part of the strategy.

The strategy will:

  • identify key priorities, activities and coordination mechanisms, incorporating Government of Canada priorities and regional office priorities as well as IBDA objectives and priorities;
  • articulate the Agency’s role and approaches along with any best practices related to supporting the attraction of FDI;
  • identify and build upon best practices and/or factors that facilitate the achievement of IBD expected outcomes; and
  • delineate any key activities or priorities that fall under the responsibility of head office (i.e. pan-Atlantic projects, including the IBDA) and regional offices.

To ensure effective implementation and monitoring of the strategy, ACOA will develop a communication plan that supports engagement and coordination, including a process to monitor the IBD strategy to ensure it that it is achieving the desired outcomes.

DG of International Business Develop-ment, in collaboration with regional DGs of Operations.

January 31, 2016

2.  To ensure that the governance framework supports evolving interests and needs, including those that emerge from the development of an IBD strategy (Recommendation 1), ACOA should review the governance structures for the IBD programming.

Strong governance structures promote clear decision making, communication and coordination of programming within the Agency. The governance review should assess roles and responsibilities related to IBD programming. It should identify mechanisms to ensure communication and shared planning between IBD and other ACOA programming, within Enterprise Development as well as with Community Development and Policy, Advocacy and Coordination.

In light of the upcoming renewal of the IBDA and building upon existing strengths and mechanisms, ACOA should also take a leadership role in the review of the IBDA governance to ensure that there is appropriate representation on the management committee and that processes related to planning and decision making are in place.

Agree

ACOA recognizes the importance of strong governance and will review the governance structures for the IBD programming. This review will:

  • include pan-Atlantic and regional BDP programming and the IBDA;
  • identify and develop mechanisms for effective communication and coordination of IBD activities, including those within Enterprise Development and those that have linkages with Community Development and Policy, Advocacy and Coordination; and
  • define the roles and responsibilities of ACOA’s head office, regional offices and, in the case of the IBDA, provincial governments, and develop processes to ensure clear decision making, planning and coordination in a decentralized structure.

 

In the renewal of the IBDA, the Agency will review management committee representation, planning processes and overall governance to ensure the IBDA governance is aligned with and supports ACOA’s decision-making processes and good governance practices.

DG of International Business Development, in collaboration with Regional DGs of Operations.

January 31, 2016

March 31, 2016

3.  To support results-based management, ACOA should build on its previous efforts to improve the IBD performance measurement strategy by reviewing and updating the program’s indicators and expected outcomes, and revising tools and processes.

The program has made important improvements to performance measurement since the previous evaluation, including the development of standardized participant questionnaires. These existing tools and processes can be built upon to streamline data collection and reporting of key performance information needed for decision making and accountability.

IBD management should continue with efforts to further streamline and develop an electronic format for the IBD participant questionnaires.

Furthermore, there is an opportunity with the current review of all ED sub-program performance measurement strategies to better align sub-program performance measurement across the Agency by identifying common expected outcomes and developing standardized indicators and data collection tools.

Agree

ACOA will review its IBD program indicators and expected outcomes to ensure it aligns with the IBD strategy. 

The IBD will participate in the current Enterprise Development Performance Measurement Strategy review exercise to ensure alignment of the IBD with Innovation and Commercialization and Productivity and Growth. This will involve clarifying and delineating between common expected outcomes and developing standardized indicators and streamlined data collection tools.

ACOA will develop and implement an electronic format for the streamlined collection of data for the IBD participant questionnaires. 

DG of International Business Development

March 31, 2016

March 31, 2016

September 31, 2015

Appendix B – Evaluation Questions, Judgment Criteria and Methods

Core Evaluation Issues and Key Evaluation Questions

Primary methods

Level of evaluation
effort

Proposed IBD Evaluation Question

Evaluation Judgment Criteria [xxxvii]

Issue 1: Continued need for programming

1.1 To what extent does the IBD sub-program continue to address demonstrable need?

1.1.2 To what extent does the IBDA continue to address a demonstrable need?

Programming related needs are still present at least to the same degree as they were five years ago.

  • Factors affecting rates of export development and investment attraction are the same as five years ago or have maintained/increased need for programming.
  • The need for export development and investment attraction is identified in existing strategies (federal and provincial), research, other documents.
  • Clients and KIs identify there is an ongoing need for IBD program activities, including marketing, outreach, training, mentoring, coordination, attracting FDI.
  • Information on the changing IBD landscape in Atlantic Canada (e.g. demographic, economical, sectorial, strategic, other programming) is collected, analyzed and considered in decision making by key stakeholders (e.g. ACOA and IBDA management).
  • KIs provide evidence of appropriate (necessary and effective) programming responses to changing landscape and needs, including changes in other programming

IBDA specific:

  • Provincial, ACOA and other IBDA stakeholders provide evidence of how IBDA programming responds to changing context and needs.
  • There is a clear rationale and need for the IBDA model/approach.

Document review

Also: KIs and client survey

LOW

Issue 2: Alignment with Government Priorities

2.1 To what extent is the IBD sub-program aligned with and does it help deliver on federal government priorities and to ACOA’s strategic outcome.

There is logical alignment between the programming, federal government priorities (e.g. Speeches from Throne, GMAP) and ACOA’s strategic outcome, priorities and strategy. The alignment is recognized, communicated and/or made explicit.

Document review

Also: KIs

 

Issue 3: Alignment with Federal Roles and Responsibilities

3.1 To what extent does the IBD sub-program align with federal roles and responsibilities?

  • ACOA’s roles and responsibilities and activities related to IBD programming, including attracting FDI and the IBDA, are aligned with the ACOA Act.
  • ACOA’s IBD programming is aligned with policy research and current policy thinking related to roles of federal government in export development and investment activities.

Document review

Also: KIs

LOW

3.2 To what extent do IBD sub-program activities, including the attraction of FDI, duplicate, overlap, or complement other programming, including other ACOA sub-programs?

3.2.1 To what extent does IBDA duplicate, overlap or complement other programming?

  • ACOA programming, including the attraction of FDI, complements rather than duplicates or overlaps that of other programming, including other federal and provincial governments.
  • There is evidence of coordination of the IBD with PG and IC; opportunities for improved coordination are known and being acted upon.
  • There is evidence that clients have received funding from other ED sub-programs.

IBDA specific:

  • IBDA activities complement rather than duplicate or overlap that of other programming, including other ACOA and provincial government programming.
  • There is a moderate to high level of satisfaction with ACOA’s support and coordination with partners among IBDA management committee members.

Document review

Also: KI; existing program data

LOW

Issue 4: Achievement of Expected Outcomes

4.1 Incrementality: What impact would the absence of IBD programming have on expected program outcomes?

4.1.1 What impact would the absence of IBDA programming have on expected program outcomes?

The impact of the absence of the sub program is similar to or greater than that observed in the previous evaluation, as reflected by:

  • ACOA clients’ perspective on whether the project/initiative would have proceeded without funding;
  • ACOA clients’ perspective on the impacts that would have occurred if there was not ACOA funding; and
  • Evidence that ACOA support influences the involvement of funding partners.

IBDA specific:

  • IBDA client’s perspective on whether the project/initiative would have proceeded without funding.

Client survey and KIs

Also: existing program data

LOW

4.2 How and to what extent has the sub program contributed to immediate outcomes (IBD) [xxxviii]:

  • Improved SME awareness of international business opportunities, including those in new and emerging markets
  • Increased international business capacity for SMEs, universities and research establishments in Atlantic Canada (i.e. knowledge, skills, culture)
  • International markets and investors have increased awareness of Atlantic Canada’s competitive advantages, capabilities and opportunities
  • Increased coordination and partnerships related to IBD in Atlantic Canada

4.2.1 How and to what extent has the IBDA contributed to these immediate outcomes?

  • There is qualitative and/or quantitative evidence that immediate outcomes are being achieved to a level at or above established targets.
  • The achievement of immediate outcomes is similar to or greater than that observed in the previous evaluation (accounting for changes in context, processes and procedures).
  • There is sufficient evidence to argue the contribution of the sub program to the achievement of immediate outcomes .

IBDA specific:

  • There is qualitative and/or quantitative evidence that immediate outcomes of IBDA are being achieved.
  • The achievement of IBDA immediate outcomes is similar to or greater that that observed in the previous evaluation (accounting for changes in context, processes and procedures).
  • There is sufficient evidence to argue the contribution of the IBDA to the achievement of immediate outcomes .  

Client survey

Also: KIs, existing program data

MED-HIGH

4.3 How and to what extent has the IBD sub program contributed to intermediate outcomes:

  • Increased number of Atlantic Canadian SMEs exporting for the first time
  • Increased export activity by SMEs in Atlantic Canada
  • Expanded foreign investment opportunities (e.g. FDI & CDIA)
  • Increased acquisition and commercialization of new technology (reported on under IC and PG)

4.3.1 How and to what extent has the IBDA contributed to these intermediate outcomes?

  • There is qualitative and/or quantitative evidence that intermediate outcomes are being achieved.
  • The achievement of intermediate outcomes is similar to or greater than that observed in the previous evaluation (accounting for changes in context, processes and procedures).
  • Clients attribute, to at least a moderate extent, the achievement of intermediate outcomes to the Agency’s IBD programming.
  • Targets set have been achieved.
  • Foreign investment opportunities (e.g. FDI & CDIA) related to IBD programming have increased since the previous evaluation.
  • There is evidence that export sales have increased since the previous evaluation.
  • There is evidence that the number of new/first time exporters has increased as a result of programming  

Client survey

Also: KIs and existing performance data

MED-HIGH

4.4 How and to what extent have intermediate outcomes contributed to the achievement of long-term outcomes (IBD):

a) Increased economic benefits for Atlantic Canada from International Markets

b) Improved growth and competitiveness of Atlantic SMEs

There is quantitative and qualitative data to support actual (or potential) contribution of immediate and intermediate outcomes toward the realization of longer-term outcomes:

  • increased revenues from international markets;
  • increased employment;
  • improved growth and competitiveness of SMEs; and
  • other economic benefits to Atlantic Canadians.

Client survey

Also: document  review, KIs and existing performance data

LOW

4.5 What, if any, unintended outcomes were achieved through the IBD sub-program?

4. 5.1 What, if any, unintended outcomes were achieved through IBDA programming?

Not applicable

KIs

LOW

4.6 What are the facilitators/best practices and barriers/lessons learned related to delivering the IBD sub-program and achieving expected outcomes? To what extent are barriers being mitigated?

4.6.1 What are the facilitators/best practices and barriers/lessons learned in the delivery of the IBDA and achievement of expected outcomes?

  • Evidence that factors facilitating the achievement of IBD expected outcomes (internal and external) are known, shared and implemented where appropriate.
  • Evidence that factors (internal and/or external) that are impeding the success of the sub-program are known and that mitigation strategies are implemented where appropriate (based on level of risk and control).
  • There is evidence that lessons learned are identified and mitigation measures are put in place.
  • Evidence that ACOA takes into consideration the findings/recommendations of previous evaluations/audits/reviews in improving delivery and considering alternative approaches related to IBD.

IBDA specific:

  • There is evidence from the IBDA management committee and others that facilitators/best practices and barriers/lessons learned are identified and that measures are put in place to respond to lessons learned.
  • There is evidence that the IBDA management committee takes into consideration the findings/recommendations of previous evaluations/audits/reviews in improving delivery and considering alternative approaches.

KIs

Also:  existing program data, client survey

MED

Issue 5: Demonstration of Efficiency and Economy

5.1 To what extent are delivery costs reasonable in relation to outcomes achieved?

5.1.1 To what extent are IBDA delivery costs reasonable?

  • Delivery costs across regional offices have remained stable or decreased since the last evaluation (accounting for changes in context, processes and procedures), and reasonably compare to one another within the context of factors affecting regional office delivery.
  • ACOA’s cost of delivering $1 in G&Cs is reasonable compared to the previous evaluation and to the delivery of other programming.
  • Delivery costs of the IBDA have remained stable or decreased since the last evaluation.

Existing program data

Also: KI and document review

LOW

5.2 To what extent are planning, performance measurement, budgeting, governance and delivery mechanisms in place that contribute to the achievement of IBD outcomes in the most efficient and economical manner?

5.2.1 To what extent are planning, performance measurement, budgeting, governance and delivery mechanisms in place that contribute to the achievement of IBDA outcomes in the most efficient and economical manner?

 

ACOA has in place resource optimization mechanisms to ensure that the most efficient and economical means are being used to administer the programming. There is evidence that:

  • governance structures and practices allow for efficient and economical delivery, including coordination of interrelated aspects of ED activities and coordination of related aspects of CD and PAC activities;
  • there is a rationale/strategy linked to expected outcomes guiding HO and regional office IBD delivery;
  • there is effective coordination and collaboration between regional offices and head office as well as among regional offices, and this leads to the selection of the best projects; shared intelligence and analysis, and strategic involvement in events and activities;
  • barriers to efficiency and economy are known and are mitigated;
  • resource mechanisms are in place that allow for the efficient allocation and re-allocation of resources;
  • there is ongoing collection and use of performance measurement data; and
  • senior management are satisfied with mechanisms for the efficient and economical administration of programming.

IBDA specific:

  • evidence that the IBDA has in place resource optimization mechanisms (as outlined above) to ensure the most efficient and economical administration; and
  • evidence that IBDA partners are satisfied with mechanisms for the efficient and economical administration of programming.

KI

Also: document review

MED

5.3 To what extent has program management considered and implemented alternative modes of delivery for the IBD sub-program?

5.3.1 To what extent have ACOA and the IBDA management committee considered and implemented alternative modes of delivery for the IBDA?

  • Evidence that program management has considered and continues to explore alternative modes of delivery within and outside the organization.

IBDA specific:

  • Evidence that the IBDA management committee considers and continues to explore alternative modes of delivery.

KI

Also: document review

LOW

Appendix C – IBDA Evaluation Findings

The IBDA is an important component of the Agency’s IBD sub-program and was included in the scope of the evaluation. Specific information needs of the IBDA management committee were considered to support the renewal of the terms and conditions of the IBDA in 2016. IBDA-specific evaluation questions were developed in consultation with the management committee and are included in the evaluation framework (Appendix B). Many findings related to the IBDA are included throughout the report and this appendix provides additional findings and further details.

The Canada-Atlantic Provinces Agreement on International Business Development

The IBDA is a pan-Atlantic partnership involving four provincial governments (New Brunswick, Newfoundland and Labrador, Nova Scotia and Prince Edward Island), ACOA and other federal government partners (the Department of Foreign Affairs and Trade Development, Industry Canada). The IBDA has four areas of focus:

IBDA funding

Funding for the IBDA is shared between the federal (70 per cent) and provincial governments (30 per cent). The financial contributions under the current agreement, signed in 2011, are provided in Table A1. Since the IBDA’s launch in 1994, the agreement has been renewed five times,[xxxix] with a total investment of $35 million from the federal and provincial governments. All partners, including non-financial partners, provide important in-kind contributions that support the achievement of positive outcomes.

Table A1: IBDA Funding Agreement, 2011-2012 to 2015-2016

IBDA Partners

Contributions ($)

ACOA (lead federal department; 70%)

7,000,000

N.S. provincial government (11%)

1,100,000

N.B. provincial government (11%)

1,100,000

N.L. provincial government (5%)

500,000

P.E.I. provincial government (3%)

300,000

DFATD (non-funding partner)

0

Industry Canada (non-funding partner)

0

Agriculture and Agri-Food Canada (observer)

0

Total contributions

$10,000,000

Source: Canada–Atlantic Provinces Agreement on International Business Development, 2011.

IBDA project administration

Project funding is administered through ACOA. The provincial governments transfer their individual IBDA contribution to ACOA on an annual basis. Through an informal agreement that goes back many years, each province retains $30,000 of their contribution to support a provincial IBD liaison position. The provincial liaisons are responsible for supporting the development and assessment of proposals as well as project implementation.

IBDA projects were funded through the International Business Development Program (IBDP) up until 2010, at which time the Trade and Investment program sub-activities were merged into the International Business Development (IBD) sub-program. From that time to now, IBDA projects are funded through ACOA’s Business Development Program (BDP).

IBDA management committee

A management committee comprised of ACOA,[xl] DFATD, Industry Canada and the four Atlantic provincial governments are responsible for the overall management of the IBDA. The management committee establishes procedures, approves projects by consensus, appoints sub-committees, evaluates projects and reports on results.

The day-to-day operations of the IBDA are overseen by an ACOA IBD sub-program manager who is responsible for liaising with the IBDA management committee and the 16 sector and market working groups. Under the responsibility of the manager is an IBDA Secretariat, which supports administrative functions, including the issuance of contracts, payments, participant questionnaire management and reporting. In addition to the manager, ACOA’s IBDA team is comprised of two program officers and two IBDA Secretariat staff. The work of the IBDA management committee is also supported by project liaison officers who are employed by each provincial partner.

IBDA expenditures

The IBDA expenditures for the period from 2008-2009 to 2013-2014 amounted to $11.4 million (Table A2). As discussed previously, projects were funded through the IBDP program until 2010 then were funded through ACOA’s BDP.

Table A2: IBDA Expenditures, 2008-2009 to 2013-2014

Fiscal Year

Transfer Payment Program

Total
($M)

BDP (IBDA)

IBDP

($M)

($M)

2008-2009

1.4

1.8

3.2

2009-2010

0.5

1.1

1.6

2010-2011

1.1

1.1

2011-2012

1.7

1.7

2012-2013

2.7

2.7

2013-2014

1.1

1.1

Total

8.5

2.9

11.4

Source: ACOA project management system (QAccess); data extracted May 2014.

IBDA project types

As shown in Table A3, over the six-year period of the IBD sub-program evaluation, there were 121 IBDA projects and most (105, or 87 per cent) were related to trade missions, mostly outgoing but also incoming. The majority of IBDA projects (106) were coded to the International Business Development Activities element, with $7.4 million in funding. The IBDA supported six projects coded to training and awareness, totalling approximately $487,000, and five related to information and intelligence gathering, totalling $290,000. Four planning and research projects received approximately $270,000.

Table A3: Number and Value of IBDA Projects by Type, 2008-2009 to 2013-2014

IBDA Elements and project type

No. of Projects

Assistance ($)

International Business Development Activities

106

7,463,295

Outgoing trade mission

81

5,522,996

Incoming trade mission

23

1,799,156

Marketing

1

74,223

Other

1

66,920

Training and Awareness

6

486,763

Training and knowledge transfer

6

486,763

Information and Intelligence

5

290,639

Planning and studies

3

186,921

Marketing

2

103,718

Planning and Research

4

270,278

Outgoing trade mission

1

94,080

Partnership and coordination

1

62,845

Planning and studies

1

70,747

Training and knowledge transfer

1

42,606

Grand Total

121

$8,510,975

Source: ACOA project management system (QAccess); data extracted May 2014

The IBDA projects were coded to particular sectors by the evaluation team (Table A4). Of the 121 projects, many involved a number of sectors (18 projects and $1.6 million in funding) for activities such as conference attendance and trade missions. The most common sectors supported were agriculture/seafood/beverage (14 projects and $1.2 million) and ocean technologies (14 projects and $1.1 million). Other common sectors were: biotech and life sciences; cultural; building and construction; information and communication technology; and education.

Table A4: Number of Projects and IBDA Assistance by Sector, 2008-2009 to 2013-2014

Sectors

IBDA Assistance

No. of Projects

Multiple sectors

1,550,513

18

Agriculture/seafood/beverage

1,168,635

14

Ocean technologies

1,108,576

14

Biotech and life sciences

871,998

11

Culture

821,816

13

Building and construction

668,657

12

Information and communications technology

573,515

10

Education

525,241

10

Environment/clean technology/renewable energy

418,354

7

Aerospace, defence and security

367,383

5

Consumer products

322,817

6

Transportation

113,470

1

Grand Total

8,510,975

121

Source: ACOA project management system (QAccess); data extracted May 2014

Targeted markets

According to project coding, IBDA projects were mostly focused on established markets (67 per cent) such as the United States and Western Europe. They also addressed emerging markets (23 per cent) such as Latin America, Asia Pacific and the Caribbean, as well as multiple markets (10 per cent).

IBDA clients

The IBDA provides financial assistance to non-profit and non-commercial organizations only, which in turn deliver support services and resources to Atlantic Canadian exporters and other stakeholders involved in international business.[xli] Over the period of the IBD evaluation, half of IBDA funding was provided to provincial Crown corporations or government departments. Almost 41 per cent supported activities led by industry associations across the four provinces. A relatively small number of IBDA clients were universities or non-profit organizations.

Table A5: IBDA Projects by Type of Client, 2008-2009 to 2013-2014

Types of Clients

No. of Projects

% Projects

Approved Funding ($)

% of IBDA Approved Funding

Provincial Crown corporation or government department

57

47.1

4,245,458

49.9

Industry associations

53

43.8

3,481,162

40.9

Other non-profit organization

6

5.0

479,374

5.6

University/college

5

4.1

304,981

3.6

Grand Total

121

100.0

8,510,975

100.0

Source: ACOA project management system (QAccess); data extracted May 2014

IBDA evaluation findings

Evaluation findings for the IBDA are largely aligned with those presented in the full evaluation report of the IBD sub-program. This section refers to relevant sections in the report while presenting specific IBDA findings. Methods used to support the identification of IBDA findings are the same as those used for the full IBD sub-program evaluation presented in section 1.2 of this report.

IBDA Relevance

IBDA is relevant to the needs of provincial, federal, industry and SME stakeholders. It is an important forum for collaboration, shared planning and the maximization of limited resources at a pan-Atlantic level. There is high satisfaction among IBDA stakeholders with the ability of IBDA to maximize limited resources and to share information.

ACOA and the IBDA management committee are aware of and are adapting programming to address changes by placing more attention on emerging markets and supporting the ability of SMEs to take advantage of new FTAs.

IBDA activities complement rather than overlap or duplicate other programming. Clarifying priorities and other differences between the Agency’s BDP funding and IBDA funding could decrease the risk of confusion among internal and external stakeholders.

As presented in section 3 of this report, there is a strong and continued need for IBD programming, including the IBDA, as challenges to international business development persist and, to some extent, have increased since the 2010 TI evaluation.

The evaluation found that ACOA and the IBDA management committee are aware of these challenges and are adapting programming to address the needs of clients. In particular, since the previous evaluation, the IBDA has put greater attention on new and emerging markets, including those aligned with new FTAs. Four new market working groups were put in place to address opportunities in China, India, Brazil and the European Union. Nearly all IBDA surveyed clients stated that the assistance met their needs to at least some extent, and 60 per cent said the assistance met their needs to a great extent.

The IBDA is complementary to other programming offered by provincial and federal governments. KIs confirmed that it offers a unique advantage by coordinating the efforts and funding of stakeholders across Atlantic Canada. However, some KIs expressed confusion about the differences between IBDA funding parameters and the pan-Atlantic projects funded through the BDP at head office.

IBDA Performance – Effectiveness

Incrementality

Like ACOA’s IBD programming, IBDA programming is incremental to the achievement of expected outcomes. According to client survey respondents, without IBDA supports, there would be negative consequences on the scope, quality and timing of projects. For every IBDA dollar provided, it leveraged $1.90 from stakeholders – up from $1.49 since the 2010 TI evaluation.

According to client survey respondents, without IBDA supports, there would be negative consequences on the scope, quality and timing of projects. On average, IBDA clients indicated a 20 per cent average likelihood of their projects occurring without IBDA assistance. In the absence of IBDA assistance, all 19 unique IBDA clients surveyed said that there would have been negative effects to their project. In particular,[xlii] they reported that the absence of IBDA assistance would have resulted in:

Financial, client survey and KI data indicate that IBDA funding influences other partners to support projects. Much like the 2010 TI evaluation, of all of ACOA’s IBD sub-program funding elements, IBDA projects leveraged funding to the greatest extent - $ 1.90, up from $ 1.49.

Achievement of immediate outcomes

IBDA programming increases SME awareness of IBD opportunities, SME capacity, and partnership and coordination related to the IBD in Atlantic Canada. The programming contributes to increased awareness among international markets and investors of Atlantic Canada’s competitive advantages, capabilities and opportunities. It contributes to at least some extent to all expected intermediate and ultimate economic outcomes though attribution is difficult.

The achievement of IBDA outcomes is facilitated by the strong collaboration among stakeholders and non-financial supports provided to SMEs that build capacity to pursue IBD opportunities.

All lines of evidence suggest that the IBDA increases SME awareness of international business development opportunities, including those in new and emerging markets:

IBDA contributed to increased international capacity for SMEs largely through supporting missions and training projects.

Very few respondents (4/19) indicated that the objective of their project was to increase the knowledge and skills of university/research institutions. Due to the limited information on university and research institutions, assessing the achievement of outcomes was not possible for IBDA specifically.

It was also not possible to determine the extent that IBDA programming contributed to international markets and investors having increased awareness of Atlantic Canada’s competitive advantage, capabilities and opportunities. However:

There is evidence that IBDA contributed to increased coordination and partnerships related to international business development in Atlantic Canada.

Achievement of intermediate outcomes

IBDA programming contributed to some extent to intermediate outcomes such as increased export activities by SMEs in Atlantic Canada. There is less evidence to support contribution to the development of first-time exporters due to limited survey and performance measurement data.

The IBDA contributed to some extent to the development of first time exporters.

The IBDA programming contributed to increased export activity by SMEs in Atlantic Canada.

Unintended outcomes

Unintended outcomes reported in the main body of the report for IBD sub-program activities apply to the IBDA programming.

Barriers and lessons learned

Barriers noted by client survey participants and KIs are consistent with those raised in the 2010 TI evaluation. Those reported in the main body of the report for IBD sub-program activities apply to IBDA programming.

Facilitators and best practices

Factors that facilitate the achievement of IBDA outcomes align with those reported in the main body of the report for the IBD sub-program overall.

Availability of IBDA performance measurement data

As reported in section 4.5 of the report, a number of improvements were made to performance measurement since ACOA’s evaluation of TI sub-activities in 2010. Notably, IBD programming, including the IBDA, implemented new standardized exit and one-year-out client questionnaires. KIs reported that IBDA performance measurement information is used to support project review processes and decision making.

Further improvements to IBD performance measurement overall support IBDA program decision making. According to a review of the performance tools and forms as well as KIs, the exit and one-year-out questionnaires are mission-centric, focus on long-term outcomes, take a long time to complete and are administered in paper format. In particular, some KIs noted that data collection is geared to the exporting of goods and sales without collecting information on broader indicators that demonstrate the extent to which programming outcomes have been reached. Gaps in data collection exist related to the work done by universities, as well as work done related to innovation and commercialization, the building of partnerships and first time exporters.

While the response rates appear strong for the exit questionnaires, mostly due to their completion tied to funding recipient payments, the response rate for one-year-out questions is very low, at only 124 completed questionnaires over a three-year period for IBD programming overall. Providing incentives and easing the collection of one-year-out data through the implementation of electronic questionnaires may improve response rates.

Performance – Efficiency and Economy

The delivery costs of the IBDA are difficult to determine due to the financial and non-financial contributions of stakeholders and the lack of IBDA specific operational expenditure reporting.

For the IBD overall, the high level of ACOA involvement in planning and delivery is a key factor in project success, and costs have remained stable since the previous evaluation.

Mechanisms are in place to allow for the efficient delivery of the programming. Further improvements could be realized by identifying and communicating clear priorities for the IBDA and reviewing the governance structure to ensure it fits with current realities.

Planning and budgeting mechanisms

IBDA planning and budgeting is mainly done through the 16 sector and working groups. As noted in the main report, the IBDA management committee has made a number of changes to improve project planning and approval processes in response to running out of funding in some years. These changes include implementing a revised project development and review procedure in which each working group prioritizes five project proposals for management committee review once a year.

While overall feedback was positive, some KIs said that the new planning and budgeting process weakens the ability to address opportunities that arise at other times of the year. It also does not prioritize the activities of any given sector or market working group over the others.

The full extent of IBDA-specific delivery costs are difficult to determine due to financial and non-financial contributions by a variety of stakeholders. Also, data from ACOA’s financial management system was tracked at the sub program level as it relates to salaries and O&M data.[xliii] Therefore, the evaluation cannot determine the exact salary and O&M expenditures associated with the IBDA. Overall, KIs reported that IBDA delivery costs had decreased over time, including through less face-to-face meetings and the use of more conference calls.

Governance

Overall, KIs viewed the governance of the IBDA positively, noting the partnership between the federal and provincial governments and the existence of sector and market working groups that engage other stakeholders. At the IBDA management committee level, decision making was done by consensus. KIs stated that decision making was facilitated by close working relationships among IBDA management committee members, which allowed for open debate when needed.

Some KIs noted that the IBDA voting membership had remained consistent over the years despite major changes in the roles and responsibilities of some federal departments and their lack of financial contributions. The large number of sector and market working groups (16) and the variability in their structure and planning processes were also identified as potential risks to the governance of the IBDA.

Delivery

Most KIs identified the high level of collaboration among IBDA stakeholders and the capacity to leverage financial resources and expertise as facilitating the efficient delivery of the programming. As reported as facilitators of programming outcomes, the programming benefits from the hands-on involvement of programming staff in trade missions to assist SMEs before, during and after such activities.

IBDA Conclusions

The IBDA findings related to relevance and effectiveness align with those of IBD programming overall. The IBDA offers a unique mechanism for pan-Atlantic collaboration that supports the achievement of positive international business development outcomes. The conclusions and recommendations include elements relevant to the IBDA programming.

Appendix D – Program Alignment Architecture

Strategic Outcome

Programs Sub-programs

A competitive
Atlantic Canadian economy
(1.0)

Enterprise Development
(1.1)

Innovation and Commercialization (1.1.1)

Productivity and Growth (1.1.2)

International Business Development (1.1.3)

Community Development
(1.2)

Community Investment (1.2.1)

Community-based Business Development (1.2.2)

Policy, Advocacy and Coordination
(1.3)

Policy (1.3.1)

Advocacy (1.3.2)

Coordination (1.3.3)

Internal Services
(1.4)

Governance and Management Support (1.4.1)

Resource Management Services (1.4.2)

Asset Management Services (1.4.3)

Source: ACOA Program Alignment Architecture, Effective April 2013.

Appendix E – IBD Conceptual/Analytical Framework

(D)

Source: ACOA, Impact Conceptual and Analytical Framework, September 2012.

Footnotes

[i] Accountability requirements are outlined in the Treasury Board’s Policy on Evaluation as well as the Directive on the Evaluation Function and the Standard on Evaluation for the Government of Canada. For small agencies, all grant and contribution spending must be evaluated every five years.

[ii] According to the Treasure Board Secretariat Directive on the Evaluation Function (Canada, 2009a), demonstration of efficiency and economy is an “assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.”

[iii] Accountability requirements are outlined in the Treasury Board Policy on Evaluation as well as the Directive on the Evaluation Function and the Standard on Evaluation for the Government of Canada. For small agencies, all grant and contribution spending must be evaluated every five years.

[iv] On March 19, 2014, the Government of Canada announced the passing of Bill C-31, legislation authorizing ACOA to assume responsibility for the direct delivery of economic development programs, services and advocacy in Cape Breton. The business and community development activities of the Enterprise Cape Breton Corporation, including the associated budget, transitioned to ACOA following passage of this legislation.

[v] These evaluation concepts are referred to as triangulation and complementarity.

[vi] While Internal Services is a program activity according to the Agency’s PAA, it does not deliver direct programming to clients; rather, it provides internal support services such as finance, corporate services, communications, etc.

[vii] For more information about the IBDA: http://www.acoa-apeca.gc.ca/eng/ibda/Pages/AboutTheAgreement.aspx

[viii] The logic model was developed by the IBD program in 2011. All Enterprise Development sub-programs were undergoing logic model revisions at the time of this evaluation.

[ix] On March 19, 2014, the Government of Canada announced the passing of Bill C-31, legislation authorizing ACOA to assume responsibility for the direct delivery of economic development programs, services and advocacy in Cape Breton. The business and community development activities of the Enterprise Cape Breton Corporation, including the associated budget, were transitioned to ACOA following passage of the legislation.

[x] A number of non-commercial projects were funded through the Atlantic Trade and Investment Partnership up until 2010. Following the sunsetting of this initiative, many non-commercial projects continued to be coded to the Partnership in ACOA’s project management system.

[xi] A 2009 Trade Audit identified weaknesses in the project management system for the monitoring of IBDP cost-sharing information and a decision was made to harmonize the delivery of the IBDA through the BDP.

[xii] For more information on the strategic review, visit: http://www.tbs-sct.gc.ca/sr-es/index-eng.asp.

[xiii] The evaluation conducted a review of the IBD sub-program O&M expenditures and identified three main categories of contracting expenditures: consultant advisory services (60 contracts at $235,882); export mentoring program in Nova Scotia (186 contracts at $570,325); and innovation in Prince Edward Island (127 contracts at $1,330,850).

[xiv] Timing differences related to the recognition of expenditures in the financial system compared to approved funding in the project database account for the variance between the total ACOA approved funding in Table 3 and the total transfer payment program expenditures shown in Table 2.

[xv] The proportion of IBD clients categorized as “other non-profit organizations” is skewed as a result of the high level of funding support provided to two separate but related organizations, which received $20 million in total funding over three years for an international security forum held in Halifax.

[xvi] The funding approved for the aerospace and defence sector includes $20 million for three international security conferences hosted in Nova Scotia over the evaluation period.

[xvii] Other sectors that received IBD funding included: cultural (5 per cent); education (3 per cent); manufacturing (3 per cent); transportation, consumer products, financial services and mining (2 per cent each); and building and construction, forestry and wood products, industrial machinery, and tourism (1 per cent each).

[xviii] Of this figure, 65 per cent indicated that ACOA’s programming met their needs to a great extent.

[xix] The IBD review was paused at the time of this evaluation, with a plan to use the evaluation findings, conclusions and recommendations to support future program planning and priority setting.

[xx] The ACOA PAA changed in 2014-2015 to reflect two sub-programs under the CD program: http://www.acoa-apeca.gc.ca/eng/publications/ParliamentaryReports/Pages/RPP_2014-15_SecI.aspx#sopaa.

[xxi] The categories of project funding in this evaluation are different from the 2010 evaluation. In the 2010 evaluation ACOA provided 50 per cent of funding, the provinces, 4 per cent, proponents, 23 per cent, and municipal and other, 23 per cent.

[xxii] Leveraging is calculated with the following formula: Total cost – ACOA assistance/ACOA assistance

[xxiii] As noted in section 1.3, there were challenges in comparing the results of the present evaluation with those of the 2010 TI sub-activities evaluation. The previous evaluation mainly used a qualitative approach to demonstrating the achievement of trade and investment immediate outcomes. However, quantitative data from client surveys showed that immediate outcomes were achieved based on Likert scale results ranging from 3.0 to 4.3 for each outcome, on a scale of 1 to 5 (where 1 = no impact; 3 = somewhat of an impact; 5 = major impact).

[xxiv] Learnsphere is a consortium of private and public training organizations that creates business opportunities for stakeholders and adds capacity to the New Brunswick training and consulting industry by securing, managing and delivering successful domestic and international projects.

[xxv] Innovation PEI delivers export-related programming such as the Export Help! Program. It has had a 20-year partnership with ACOA, EDC, Canada Business Network, Invest PEI, Agriculture and Forestry PEI, Fisheries, Aquaculture and Rural Development, Agriculture and Agri-Food Canada, and Foreign Affairs and Trade Development Canada.

[xxvi] As noted previously in Section 1.3, there were challenges in comparing the results of the current evaluation with the 2010 TI evaluation.

[xxvii] This indicator was modified in 2011-2012 to the number of SMEs expanding their international sales, which includes new exporters, exporters to new markets and increased sales to existing markets.

[xxviii] 2008-2009: 74 achieved; target 35. 2009-2010: 76 achieved; target 35. 2010-2011: 105 achieved; target 48.

[xxix] Full-time employment means an average of 30 hours a week or more.

[xxx] Comparison to the 2010 evaluation is not possible due to changes in the PAA and different methodological approaches. In the previous evaluation, of 146 trade clients, 119 (82 per cent) said they were exporting, with export sales estimated at $661 million; 141 companies reported export revenues totalling $1.1 million in the year of the interview; 141 companies reported revenues totalling $1.1 million in the current year. In the survey administered in the fall of 2014, clients were asked to report revenues for the most recent year. Therefore, it is assumed clients reported on the 2013-2014 fiscal year, or if they use the calendar year, on 2013.

[xxxi] Clients were asked to report on the most recent fiscal year. Since the survey was administered in the fall of 2014, it is assumed clients reported on the 2013-2014 fiscal year, or if they use the calendar year, on 2013.

[xxxii] A single client reported $75 million of the $103.9 million in total increased revenues.

[xxxiii] A small number of survey respondents reported other barriers such as weather, difficulty in attracting qualified staff, large/rural scope, limited production capacity and short project deadlines.

[xxxiv] On a scale of 1 to 5, where 1 is not at all, and 5 is to a great extent.

[xxxv] According to the Treasure Board Secretariat Directive on the Evaluation Function (Canada, 2009a), demonstration of efficiency and economy is an “assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.”

[xxxvi] Budget 2010 announced that operating and salary budgets would be frozen at their 2010-2011 levels for fiscal years 2011-2012 and 2012-2013. ACOA also implemented a series of measures during those fiscal years to address reductions to the general operating costs. These included caps on travel, hospitality, conferences and professional services. Budget 2011 identified savings of $15.2 million by 2014-2015 for ACOA. A review of ACOA’s activities allowed the Agency to identify ways to consolidate various functions, reduce the duplication of work, and eliminate non-priority activities without impacting the Agency’s ability to deliver on its core programs.

[xxxvii] Coverage of judgment criteria will depend on the existence of available data and level of effort associated with the collection of additional data. Judgment criteria may be modified or removed if the level of effort outweighs the value to decision making. In most cases, all judgment criteria will assist in addressing the IBDA-specific questions as well as the general IBD sub-program questions.

[xxxviii] A revised IBD sub-program logic model is currently under development and changes to expected outcomes are pending. The expected outcomes that appear in the evaluation framework have been modified from the current IBD logic model at the time of TOR development, based on discussion with program management.

[xxxix] The IBDA was renewed by partners in 1997, 2000, 2005, 2010 and 2011.

[xl] ACOA has two votes on the committee; one is held by ACOA’s head office and the other is shared by the regional offices which rotate a vote at the IBDA management committee at each meeting.

[xli] According to the IBDA agreement, Schedule B, eligible funding recipients are not-for-profit entities, including provincial governments, provincial Crown corporations and municipal governments.

[xlii] The respondents were allowed to select all that apply, therefore, the tabulation of results will show a total percentage beyond 100.

[xliii] The IBD salaries have increased from $3.7 million in 2008-2009 to $4.9 million in 2013-2014, while the operational and maintenance expenditures have decreased from $1.9 million in 2008-2009 to $600,000 in 2013-2014. Delivery costs decreased over the evaluation period; for every dollar in funding, the cost of delivering the IBD was $0.27, down from $0.29 as indicated in the 2010 TI evaluation.

Page details

Date modified: