Evaluation of the Canada Brain Research Fund Program 2016-17 to 2020-21

Table of contents

• Executive summary
• 1.0 Evaluation purpose
• 2.0 Evaluation scope, approach, and design
• 3.0 Program profile
• 4.0 Findings: alignment with priorities
  • 4.1 Current and projected need for brain health research
  • 4.2 Addressing research priorities
    • 4.2.1 Commitment to research in the ecosystem
    • 4.2.2 Addressing identified research and policy priorities
  • 4.3 Added value to the brain research ecosystem
• 5.0 Performance: achievement of expected outcomes (leveraging resources, building partnerships, and knowledge translation)
  • 5.1 To what extent have the outcomes been achieved?
  • 5.2 Translation of research to clinical application
• 6.0 Findings: Delivery model
  • 6.1 Best practices and lessons learned
    • 6.1.1 Equity, Diversity, and Inclusion
    • 6.1.2 Length of agreement
    • 6.1.3 Funding parameters
    • 6.1.4 Performance measurement
• 7.0 Conclusion
• 8.0 Recommendations
• 9.0 Management Response and Action Plan
• Appendix 1: Comparative analysis chart
• Appendix 2: Impacts of two-year funding for contribution agreements
• Appendix 3: Evaluation methods and limitations
• Endnotes

List of acronyms

ALS

Amyotrophic lateral sclerosis

BCF

Brain Canada Foundation

CAD

Canadian Dollars

CQDM

Consortium de recherché biopharmaceutique

CIFAR

Canadian Institute for Advanced Research

CIHR

Canadian Institute of Health Research

EDI

Equity, Diversity, Inclusion

GoC

Government of Canada

HC

Health Canada

INMHA

Institute of Neuroscience, Mental health and Addiction

IPSC

Induced pluripotent stem cells

ISED

Innovation, Science, and Economic Development

KT

Knowledge Translation

PIP

Performance Information Profile

PM

Performance Measurement

Executive summary

Program profile

Through various budgets, the Government of Canada has committed a cumulative total of $200 million for the Canada Brain Research Fund Program within Health Canada to support Canadian neuroscience research, and to help the medical community better understand the brain and brain health. This program supports research with the greatest potential to lead to scientific discoveries that advance therapies and approaches to improve the health and the quality of life of Canadians at risk of, or affected by, brain health conditions. The Brain Canada Foundation (the BCF), the sole recipient of program funding, works in partnership with a range of donors from across the private and charitable sectors to provide competitively awarded funding for research across Canada.

The federal funding model requires the BCF to raise 1:1 matching funds from donors outside the federal government in order to support research projects. Examples of BCF's donors and partners include research institutions, health charities, and corporations.

The BCF aims to shape its research priorities by engaging with the neuroscience community and bringing research stakeholders together to discuss and advance key brain health issues and opportunities in the health sector. Its flagship programs, which include team, platform, and capacity building awards, are designed to fill gaps identified by stakeholders for increasing research capacity and strategically advancing the prevention, diagnosis, and treatment of brain health disorders.

Conclusions

There is a continued need for research in the area of brain and neurological health, given that the current financial burden of disease associated with brain and neurological health conditions and disorders reaches into the tens of billions of dollars annually, and is only increasing over time.

Through the BCF, the Program addresses research and policy priorities identified by brain research stakeholders, either as a shared priority or by engaging with research partners to conduct research in specific priorities of interest. Over the course of the period covered by the evaluation, the BCF has more than doubled the federal funding provided through donations leveraged from private and non-governmental sources, surpassing the 1:1 matching model.

The BCF is a well-established organization and its knowledge and expertise in the brain-related health sector have helped to connect researchers with funders, as well as create partnerships and collaborations among researchers with diverse scientific and operational backgrounds. These relationships continue to support "high risk, high reward" research and contribute to enhancing the brain-related research ecosystem in Canada:

• Funding partners have indicated that their collaboration with the BCF has helped them support quality research they may not have otherwise pursued, especially when they were lacking scientific or operational expertise in the sector. In addition, the BCF has met or exceeded most of its performance targets related to capacity building, demonstrating that the Program is contributing to the development of researchers within the ecosystem.
• Some research funded through BCF partnerships is starting to be translated into intellectual property, demonstrating that intermediate outcomes are being achieved.

As noted in key informant interviews, Health Canada's Canada Brain Research Fund Program has developed a positive and constructive relationship with the BCF, resulting in a well-managed contribution agreement. Given the maturity of the BCF, a review of the terms and conditions of the contribution agreement between the BCF and Health Canada could reduce process irritants, which will further support success, as well as measure and communicate impact more effectively in support of decision making.

Expanding the BCF's partnership reach internationally could result in more opportunities to fund research that increases capacity for brain research in Canada, promotes access for Canadian researchers to research platforms in other countries, and shares knowledge generated by Canadian-produced research more broadly. As such, supporting international collaborations with BCF-funded researchers could both increase the expertise available to Canadian brain research efforts, and broaden the dissemination and use of knowledge generated by BCF funding partnerships at the international level.

Recommendations

The Canada Brain Research Fund Program's 1:1 matching funding model has added value to the brain research ecosystem, and has provided support for addressing the financial burden of brain and neurological diseases in Canada. The following recommendations focus on areas that could support continuous improvement in this area:

Recommendation 1: Where possible, consider any modifications to program parameters or their interpretation in order to enable greater opportunities for collaboration and potential international research partnerships.

The current terms and conditions of the Program, as well as the contribution agreement have presented challenges to creating international partnerships. This is an important matter to resolve as forging these international partnerships has the potential to increase capacity for brain research in Canada, promote researcher access to research platforms, and to share knowledge generated by Canadian-produced research more broadly.

Recommendation 2: To increase the likelihood of outcome achievement and to measure and communicate the Program's impact for Canadians and support decision-makers more effectively, consider consulting with partners, researchers, and institutions in the brain-related research ecosystem to identify optimal approaches to performance measurement and knowledge translation that recognize:

1. a. the nature of a networked research ecosystem; and
2. b. the complex nature of multi-disciplinary research.

Throughout the brain and neurological research ecosystem, challenges have been identified in capturing performance data that is meaningful to funders and administrators. As well, optimal approaches for translating research knowledge to achieve intermediate and ultimate impacts are not generally agreed upon, nor applied throughout the ecosystem. Notably, there is a lack of consensus as to how to address these challenges.

1.0 Evaluation purpose

The purpose of the evaluation was to examine the extent to which Health Canada's Canada Brain Research Fund Program (the Program) and Brain Canada Foundation (BCF) have achieved expected outcomes. Although the scope of the evaluation includes BCF activities during the period of 2016-17 to 2020-21, case studies that were initiated prior to this period were taken into consideration, owing to the long time that it takes for funded research to be translated into measurable outcomes.

This was a scheduled evaluation, as per Health Canada's Five-Year Evaluation Plan 2021-22 to 2026-27, in accordance with the requirements outlined in the Financial Administration Act.

2.0 Evaluation scope, approach, and design

The scope of the evaluation covered the period ranging from April 2016 to March 2021, and included an examination of the Program's contribution to support BCF objectives. The evaluation focused on BCF activities related to research funding and the funding model itself. This entailed an examination of the Program's outcomes, including the intermediate and long-term impacts of the BCF's funded research. The evaluation used multiple lines of evidence, such as literature and document reviews, performance data, and key informant interviews, to ensure triangulation of findings (see Appendix 3 for a detailed methodology, limitations, and mitigation strategy.)

3.0 Program profile

The Program is a named-recipient contribution program that provides support for neuroscience research through funding for the BCF. The BCF, which is the sole grants and contributions recipient of the Program, works in partnership with a range of donors from across the private and charitable sectors to provide competitively awarded funding for research across Canada. Program funding for the BCF supports the full spectrum of brain and mental health research, with a focus on "high-risk, high-reward" research through innovative collaborations. These types of projects come with a higher degree of risk, in that they propose new approaches that push the boundaries of current research paradigms. However, they also have the potential to result in significant discoveries and advancements.

The BCF is a national charitable organization that raises funds to foster advances in neuroscience discovery research, with the aim of enhancing the scientific community's understanding of brain health and improving care for those affected by brain health conditions, including neurological and mental health issues. The BCF receives annual contributions from the Program, as determined through federal budget allocations. Since 2011, this has totalled $130M in federal investments, for a total matched investment of over $260M. An investment by HC of $200M is projected by 2024-25. The BCF is required to raise funds to match, on a 1:1 basis, the federal investment in the Fund.

The BCF delivers its mandate through three main program lines, which reflect the types of funding competitions that award funding:

1. Team awards

Team awards support multidisciplinary research teams and aim to accelerate novel and transformative research that will change the understanding of nervous system function and dysfunction. These awards represent approximately 47% of funding disbursed over the period of the evaluation, with the average duration of each award being three years or more.

2. Platform support awards

The Platform Support awards are intended to facilitate and accelerate research by funding the maintenance and operation of major existing research platforms, providing the national or regional technical capability to multiple neuroscience investigators. These awards represent approximately 39% of funding disbursed over the period of the evaluation, with the average duration of each award being three years or more.

3. Capacity building awards

To promote the next generation of researchers, capacity-building awards support outstanding early and mid-career researchers undertaking original research related to brain health. These awards represent approximately 14% of funding disbursed over the period of the evaluation, with the average duration of each award being two years or more.

As well, the BCF undertakes a variety of Knowledge Translation activities in support of its mandate.

4.0 Findings: alignment with priorities

4.1 Current and projected need for brain health research

There is a continued need for research in the area of brain and neurological health, given the current and growing burden for brain-related diseases and disorders. With respect to mental health specifically, there is a strong link between poverty and addiction issues, along with neurological conditions affecting mental health. Brain and neurological health disorders include mental health diseases and disorders, neurodegenerative disorders such as dementia, Alzheimer's, Amyotrophic Lateral Sclerosis (ALS), Multiple Sclerosis, brain and spinal cord injuries, brain cancers, strokes, Parkinson's Disease Syndrome, trauma, and stress disorders, among others. Canadians are increasingly living with or caring for someone with one or more of these disorders, and the systemic and individual costs associated with prevention, treatment, and care are increasing over time. Estimates of the burden have not been completed consistently across all disease and disorder types for brain and neurological health disorders; the last time such an exercise was undertaken was in 2013^{Footnote 1}, and the estimated total economic burden was calculated at $193B annually.

This burden is expected to increase. The diagnosis of mental health disorders, including mood disorders, such as depression, and psychotic disorders, among others, increased by 55% between 2007 and 2017. In 2016, it was estimated that the prevalence of Canadians living with a mental health disorder would increase between 2011 and 2041, from 4 million Canadians to 4.9 million.

With respect to neurological disorders, the growing population of aging Canadians is expected to increase the prevalence of neurodegenerative disorders, including dementia. By 2031, 25% of the Canadian population will be seniors, a significant milestone in this context, as aging is the most important risk factor associated with dementia. In 2016-17, more than 432,000 Canadian seniors (7%) aged 65 years and older were living with diagnosed dementia. Furthermore, as this number does not include those under age 65 who may have a young-onset diagnosis, nor those that have not been diagnosed, the true scope of dementia in Canada may be somewhat broader.

The increasing number of Canadians living with neurodegenerative conditions such as dementia creates a societal challenge, resulting in drastically increased costs for society, governments, and individuals. It is estimated that by 2036, 62% of the Canadian health care budget will be dedicated to senior care. Other rising costs associated with neurological/ and neuromuscular disorders include strokes ($3.6 billion) and Parkinson's Disease ($1.2 billion); other diseases and disorders, such as brain cancers, spinal cord injuries, and ALS, have had cost burdens calculated at a patient-level only, rather than at a systems level.

The BCF is in a position to help address this burden by being a focal point for investment in public, private, and voluntary research across the brain research ecosystem.

4.2 Addressing research priorities

4.2.1 Commitment to research in the ecosystem

Owing to the growing burden of disease associated with brain and neurological diseases, the Government of Canada (GoC) has made commitments to the brain and neurological health ecosystem to address the spectrum of issues in this area. Investments are being made in key areas, such as mental health and neurodegenerative diseases, to foster research within the ecosystem. For instance, the GoC has allocated specific amounts to support brain and neurological health research: in Budget 2021, $996M over five years was allocated to address mental health issues, and in Budget 2019, $50M over five years was allocated for the implementation of the national Dementia Strategy. From the perspective of addressing these GoC priorities, the BCF has funded research into the specific areas of mental health and neurodegenerative diseases, complementing the research interests of the GoC.

Another GoC commitment is the ongoing funding of the Canadian Institutes for Health Research's Institute of Neurosciences, Mental Health and Addiction (INMHA), which is finalizing a mental health strategy to address a gap in mental health research, and the CIHR Institute on Aging, which includes a focus on dementia and other neurodegenerative disorders. HC also continues to fund the Mental Health Commission of Canada ($14.25M per year) and the Canadian Centre on Substance Use and Addiction through contribution agreements. The Centre for Aging and Brain Health Innovation was funded with $123.5M over five years, from 2015 to 2020; however, funding has not been renewed. As well, the 2018 document "A Common Statement of Principles on Shared Health Priorities," focuses partly on mental health as a priority area of federal funding to provinces and territories to address an increasing need in that area.^{Footnote 2}

It has been reported that Canada's science funding is the second lowest in the G7^{Footnote 3}. Only 7.2% of its health budget is dedicated to mental health care, which may be considered as proportionally low in the context of the magnitude of the burden of disease that this represents in Canada^{Footnote 4}. Finally, as noted by the Canadian Brain Research Strategy, there is currently no national policy or coordination of brain and neurological health research to achieve collective goals within the ecosystem.^{Footnote 5}

4.2.2 Addressing identified research and policy priorities

Key informant interviews, document review, and comparative analysis (Appendix 1) show that owing to its national mandate to fund research while working within the brain research ecosystem, the BCF's research priorities align with those of other groups operating within the that same ecosystem. This is because BCF establishes its research priorities in consultation with donors, partners, researchers and institutions. As well, the terms and conditions of the BCF's contribution agreement provide it with the flexibility to fund projects outside of its stated priorities, to meet emerging areas of research.

Importantly, the BCF supports the research conducted by other stakeholders in the research ecosystem through its mandate to match funding with partners on projects of mutual priority. In this way, the BCF is able to address the vast majority of research priorities in the ecosystem directly through co-funding. Furthermore, owing to the open call structure of its competitions, projects reflect the research interests of researchers and their institutions, which ensures that research interests and priorities are complementary or aligned between the BCF, its co-funding partners, and researchers who are awarded funding.

The only gaps in research priorities that were identified are priorities that address system issues, such as patient-oriented research, establishing sector-wide research and policy standards, or sector-wide strategic alignment within the ecosystem. Notably, addressing these issues would require sustained and broad-based engagement between willing key partners that would need to include a range of government and non-governmental entities. Otherwise, BCF funded research is inherently complementary to the research of its partners, as they only engage in partnerships in areas of mutual interest. This adds value to the ecosystem by ensuring that research priorities are addressed.

Research in brain health, especially in mental health and neurodegenerative disorders continues to be a funding priority for the GoC and HC. Through the BCF, the Program is addressing identified research and policy priorities across all brain research stakeholder groups, either as a shared priority or by engaging with partners to conduct research in that area. A small number of gaps have been identified through key informant interviews and document reviews.

4.3 Added value to the brain research ecosystem

Owing to its national mandate to conduct research while working with an extensive network of partners, the BCF adds value through its matching funding model and expertise in the sector. In key informant interviews, BCF funding partners noted that the value the BCF brings to the research ecosystem is largely due to its matching funding model and its knowledge and expertise in the sector. Among those partners operating in the not-for-profit sector and those representing private foundations, some key informants stated that they engaged in a funding partnership because the 1:1 matching funding model permitted them to undertake research funding that they would not have otherwise been able to. Key informants indicated that these funding partnerships helped them undertake great research, enabled in-depth research review, provided infrastructure for researchers, gave them access to many people with experience and expertise, as well as to the resources of other institutions. Also, key informants who were researchers noted that the leveraged, matched funding model allowed them to undertake more complex research projects than they could without the additional matched funding. Furthermore, other key informants noted that, although they were interested in funding research in the area of brain and neurological health, their lack of knowledge of the research ecosystems was a barrier to accessing it; they did not know who to partner with, nor how to navigate the research area. However, they found that the expertise and experience of the BCF gave them the confidence to create a meaningful funding partnership with a trusted organization. As one representative from a not-for-profit partner remarked:

"… and because we are not mental health experts and we do not want to do RFPs and run our own selection processes, we see them (the BCF) as a trusted partner with the expertise in their scientific advisory groups, and their board, and all of their different groups to actually ensure that what we fund is the best."

All of the key informants who were recipients of Team awards identified the leveraging of matching funding with the BCF as being the awards' main value. Furthermore, the recipients of Platform awards also noted that the leveraging of funding was the main value to them, as platforms require long-term investment with multiple partners to achieve sustainability over time. One key informant from a research institute provided an example of how the initial matched $1.2M funding from the BCF from 2015 to 2018 to create a research platform had attracted an additional $17M in funding from 2015 to 2020. More specifically, BCF funding contributed to the creation of the research platform, which drew interest from other funding sources, such as the pharmaceutical industry. The initial matched funding has a number of benefits, such as the means to hire additional researchers, including students. A further benefit identified by key informants is that both Team and Platform funding partnerships play a role in capacity building, as projects and platforms require teams of researchers, including early career researchers and students, to be successful.

On the subject of early career researchers, key informants also noted that the BCF Future Leaders awards in particular are unique within the funding ecosystem. Among the benefits identified, it was noted that this funding is dedicated to building capacity specifically in the area of brain and neuroscience research. To put the BCF awards into context, the following chart presents the only known neuroscience-specific awards for early career researchers:^{Footnote 6}

Table 1: Neuroscience awards in Canada by size and number

Award	$ Amount and # of Grants (where available)
The BCF – Future Leaders in Canadian Brain Research Program	$100,000 for each of 20 awards, $2M total
CIHR – Brain Star awards (with the Canadian Association for Neuroscience)	$1,500 to 15 trainees who published research relevant to CIHR-INMHA's mandate in 2020.
Canadian Association for Neuroscience	Conference registration and $1000 for one candidate

Not only is this dedicated funding for early career researchers significant, it was also noted by key informants that dedicated funding for brain and neurological health researchers and researcher teams is especially valuable since the overall number of researchers being funded by CIHR has been in decline over the past several years.^{Footnote 7}

Although the vast majority of key informants believed that the leveraged funding model brought value to the ecosystem, three key informants believed that the BCF did not bring new funding into the brain research sector. Rather, they felt that the funding leveraged from other research partners would have entered into the ecosystem regardless of the BCF's matching model, although they were not specific as to how this funding would have made its way into the brain health ecosystem without the BCF. However, even these informants recognized that all funding dedicated to brain research is welcomed.

Another identified value of the BCF was its dedication to funding "high-risk, high-reward" research. For not-for-profit and private foundations, this approach was attractive to them as they sought not only to leverage their funding, but also to ensure that the research they fund achieves the highest level of impact possible, even in new or emerging areas where this level of impact was not guaranteed (the BCF approach matched the risk profile for research of its co-funders). Key informants from research teams noted that, apart from the BCF Team awards, there was little funding for high-risk research in the brain and neuroscience sector. Finally, the comparative analysis illustrates that the BCF focus on "high-risk, high-reward" research is complementary to the comparator institutions' focus on health systems, clinical and commercial translation, and public health aspects of research.

5.0 Performance: achievement of expected outcomes (leveraging resources, building partnerships, and knowledge translation)

5.1 To what extent have the outcomes been achieved?

5.2 Translation of research to clinical application

With an average time horizon of 17 years to translate basic research into clinical applications and other intermediate impacts, BCF case studies are demonstrating promising progress in achieving these impacts.^{Footnote 15} Below are three case studies that are demonstrative of the long-term horizon of achieving outcomes for the BCF-funded research. In total, eight case studies were examined for this evaluation^{Footnote 16}, including case studies of research awards that precede the evaluation period, reflecting the long time horizon of translating research into applications.

Table 4: Select BCF Case Studies

Project	Help for Kids with Injured Brains Led by a team from the Hospital for Sick Children in Toronto, which received a 2012 grant totalling $1.5M. The purpose was to determine whether metformin, a type 2 diabetes drug, and physical exercise could stimulate the recruitment of healthy brain cells, and in doing so, promote repair in the injured brain. To enhance the translational potential of their research, the studies were conducted in both mice and humans, with a particular focus on brain repair in children with acquired brain injury.	Partnership for Stroke Recovery Advances in stroke treatment have increased survival, but have resulted in more people living with chronic disability. Research emphasis has shifted to treatments to enhance recovery. Community-based exercise programs for those living with stroke have been identified through prior research as being a promising approach to treatment. This project is intended to scale up the implementation of sustainable and evidence-based community exercise programs for those living with stroke, and to measure the impact of uptake. The initial grant award, in partnership with the Heart and Stroke Foundation and the Canadian Partnership for Stroke Recovery, was $1.4M.	Early Years Program The Early Years (EY) Program is a project funded by Brain Canada in partnership with the Martin Family Initiative (MFI) that aims to improve outcomes for pregnant Indigenous women and their children. The Early Years program is providing families with the tools to uplift their children in ways that promote resiliency and attachment, foster early language development, and establish overall wellness. The initial grant amount provided to the Martin Family Initiative by the BCF was $3.5M.
Immediate Outcomes	In 2016, the research led to a number of notable findings from both their animal and human studies, resulting in 13 publications in several high-impact journals, including Science, Cell Stem Cell, and Neuron. Their findings that metformin could enhance brain repair and functional recovery in mice, and exercise could affect brain structure and cause a significant increase in cognitive function in children with acquired brain injury, set the stage for their human work with metformin.	Involving more than 300 Canadians in 10 cities, the study found that using technology to delivery post-stroke therapy exercise programs can be as effective as traditional care. This is especially important for delivering post-stroke therapy to people in rural and remote parts of Canada.	The project has the potential to demonstrate how to effectively support Indigenous child well-being in a holistic and outcome-driven way. Preliminary qualitative and quantitative data from the first 18 months of this project already shows evidence of participant and community engagement, social and health service navigation and access, as well as parental support and mitigation of stress.
Intermediate Outcomes	In 2019, three years following the completion of the grant, findings were published in the open-access journal, Science Advances, showing that metformin promoted cognitive recovery in a mouse model of neonatal stroke, and had sex-dependent effects. The research continues to make an impact: currently, three clinical trials have stemmed from the original discovery, including one exploring the impact of metformin on recovery in children with acquired brain injury caused by tumour radiation as part of treatment. In 2021, the Multiple Sclerosis Society of Canada announced $400,000 in funding to support a pilot clinical trial to investigate the use of metformin as a therapy for children and young adults with multiple sclerosis (MS). In partnership with Stem Cell Network and Ontario Institute for Regenerative Medicine, this investment increases the funding for that trial to $1 million. Although the BCF is not involved in this project, this demonstrates how the knowledge is now being taken further toward clinical treatments.	These insights have helped inform the rapid increase of tele-rehabilitation during the pandemic. As noted by a key informant, two collaborating studies are currently underway (with one in its initial phase) and four more have been identified. All studies will continue to test new approaches, therapies, therapeutics, and technologies to improve stroke recovery. These four studies will start once research resumes post-pandemic.	Early successes have been recognized by other Indigenous groups, with plans to expand the program to one community in Nunavut in 2020, as well as to four communities in Yukon and nine in British Columbia in 2021. So far, additional funding of $12M has been raised.

6.0 Findings: Delivery model

6.1 Best practices and lessons learned

6.1.1 Equity, Diversity, and Inclusion

The BCF has been incorporating Equity, Diversity, and Inclusion (EDI) best practices into its processes through policy actions, which are having an impact on the diversity of its applicants. Its approach to EDI practices is in line with acknowledged best practices of the Canada Research Coordinating Committee; however, it should be noted that EDI is still an emerging administrative practice, and it will be necessary for the BCF to remain informed of emerging developments.^{Footnote 17}

In 2018, the Government of Canada began to prioritize equity, diversity, and inclusion (EDI) practices in the broader research ecosystem through the Canada Research Coordinating Committee.^{Footnote 18} Furthermore, the goals for EDI practices have been articulated through CIHR's work in its Sex and Gender-Based Analysis Action Plan, and through Health Canada's Sex and Gender Action Plan^{Footnote 19}. Overall goals for the EDI action plans entail supporting equitable access to funding, promoting EDI considerations into research design, increasing equity and diversity in research teams, collecting appropriate performance, and undertaking the data analyses needed to include EDI considerations into decision making. Another best practice not included in the Action Plans is promoting institutional accountability. This practice is a characteristic of the Canada Research Chairs Program.^{Footnote 20}

Since 2018, the BCF has taken specific actions to address EDI concerns at the pre-award, peer review, and post-award phases of its program competitions. In 2018, 75% of BCF-funded programs had EDI considerations incorporated into their processes; by 2019, this had increased to 100%.

The longer-term changes resulting from the internal policies have not yet been captured through performance measures, which have only recently been developed; however, data is already being collected and reported.

The following table captures acknowledged best practices from CIHR's Action Plan for EDI, as well as the Canada Research Chairs program accountability practices, and the actions that the BCF has undertaken to address these practices:

Table 5: Comparison of EDI Best Practices and the BCF EDI Best Practices

Best Practice	BCF EDI Practices
Support equitable access to funding.	At the pre-award stage, Requests for Applications for funding, Letters of Intent, and the Full Application all state support for EDI principles, and the review processes for these applications have been rendered gender neutral.
Promote EDI considerations into research design.	Recent competition applications contain EDI considerations in application requests, including how EDI is incorporated and considered as part of the proposed research design.
Increase equity and diversity in research teams – ensuring that research teams are diverse.	Applications state support for EDI and ask for equity identification of proposed team members.
Peer review	Efforts to make review panels as diverse as possible, including criteria such as reviewer expertise and gender. The review process reflects EDI considerations in the Criteria for Assessment, and peer reviewers are guided through addressing unconscious bias in the review process.
Institutional accountability	EDI performance measures reported to the Program in the PIP.

6.1.2 Length of agreement

Starting in 2017, the funding period for the BCF was reduced from five years to three years; this was further reduced to two years in subsequent contribution agreements. The change in the duration of the contribution agreement was in anticipation of the creation of the Strategic Science Fund (SSF), which was announced in Budget 2019 as the new approach to funding third-party science and research organizations. Although the fund launched in summer 2021, its first disbursements are not expected to begin until 2024-25.

The current two-year agreement length has created administrative challenges that were not present under the five-year agreements. As shown in Appendix 2, challenges have arisen in allocating funds for programs, owing to the one-year period that it takes to approve a new contribution agreement and the one-year period that it takes to design and implement a research program partnership. To some extent, the Program and the BCF have been able to work around this limitation. For instance, staggering program and partner matched funding so that the BCF's funds are spent for the first two years of a research program's duration, and then donor funds are subsequently spent. However, the two-year funding timeline may still lead to lapsed funding. Another consequence of the two-year agreements is that the length of project duration appears to have been reduced. As demonstrated in Appendix 2, this is reflected in program data, in which the length of research programs has declined since 2017; partnerships lasting five years or more are no longer being undertaken, suggesting that projects of a certain scope are no longer being considered for funding.

Furthermore, as revealed in interviews and other documents, the two-year horizon has limited the opportunity to pursue some partnerships with private foundations or registered charities, owing to their legal obligations under the Income Tax Act that restrict how their donations can be used (i.e., preventing the donors from donating in the outer years of the agreement).

Recent findings and recommendations from evaluations of other science-based research-related initiatives, highlighted below, have noted that even a five-year time horizon for funding in the research environment may not be adequate to support either fundamental or translational research properly through to the knowledge translation phase of a project. This also has implications from a performance measurement perspective, as the shorter timeframe of projects means that data collection and reporting focus on outputs and immediate outcomes, rather than intermediate and ultimate outcomes. This presents a clear challenge in measuring these impacts and assessing the long-term successes of the Program. For example:

• In 2020, the evaluation of the Genome Research and Development Initiative, notes that the "Five-year lifecycle of SPPs [Shared Priority Projects] is too short to demonstrate socioeconomic outcomes of the scientific research," and therefore poses challenges for long-term measurement and translation^{Footnote 21}; and
• In 2019, an evaluation of the Futurpreneur research program concluded that "ISED should examine longer-term funding contribution options to support program stability and Futurpreneur's progress towards intermediate and ultimate outcomes."^{Footnote 22}

6.1.3 Funding parameters

Key informants have noted that the funding mechanism has been beneficial for the BCF. However, there are improvements that can be made to promote the BCF's successes further. As discussed previously in this report, the matching funding model is particularly appealing to its range of funding partners and donors, including those from the private sector and not-for-profit foundations. As noted in key informant interviews, it is this matching funding that helped to attract them to the BCF and the sector. In particular, the matching model is also a way of supporting other not-for-profit organizations to undertake research funding on a larger scale than they could have through their own funding.

Key Informants in this study note that Program funding provides a measure of stability to the BCF.

Key informants have also noted that the funding provided by the Program provides a measure of stability to the BCF. First, as noted by researchers from other institutions, the Program's contribution represents "fenced funding" specifically dedicated to brain research in Canada. This ensures that there is a minimum level of brain-related research, and its related benefits to capacity building, available annually to the ecosystem.

Although refinements to the Program's parameters have been made over the past several years, two notable areas for improvement include:

• International collaborations: Although establishing funding partnerships is a priority for the BCF to build capacity, leverage resources, and translate its knowledge, progress in this area has proven challenging when it comes to partnering with international organizations and their donors. This is due to a lack of clarity in the Program's terms and conditions that prevents the allocation of funds for research conducted outside of Canada. The BCF also faces challenges with tracking and monitoring these funds with the same standards and practices as applied to Canadian donors. As noted by key informants, including researchers, forging international partnerships has the potential to increase capacity building for brain research in Canada, to promote researcher access to research platforms, and to share knowledge generated by Canadian-produced research more broadly.
• Knowledge translation: Although the BCF has partnered to fund some knowledge translation (KT) programs, and does engage in certain KT activities, these are ad hoc efforts and are not organized around a central approach or strategy. KT, however, is a key factor that supports the achievement of higher-level impacts, such as the development of intellectual property from basic research. During key informant interviews and in the comparative review, it was noted that a few other research institutions active in the brain sector and broader research sectors specifically fund a KT business line. These business lines exist to assist systematically with the translation of research into clinical and patient-oriented results or intellectual property. An expanded or more systematic approach to KT by the BCF could support the achievement of wider KT impacts. An initial approach to adopting a more formal KT strategy could be to embed KT throughout BCF research competitions and corporate processes. Another approach may be to fund a separate business line. However, current program parameters only allow 10% of funds to be dedicated to operational costs, so additional funding or changes to program parameters may be needed to fund a dedicated business line. This approach of funding a KT business line, however, may not be attractive to the BCF matching funding partners, which engage with the BCF to fund basic research.

Thus, certain elements of the Program's terms and conditions for the funding agreement, especially the short duration and provisions for collaborations between the BCF and potential partners, may have affected the BCF's ability to achieve their desired outcomes. Apart from these challenges, the matching funding model has helped the BCF to add value to the brain research ecosystem.

6.1.4 Performance measurement

Interviews with key informants from other government agencies, academic and other institutions, foundations, and not-for-profit organizations have demonstrated that the current approach to performance measurement and reporting undertaken by the Program with the BCF is appropriate in scope, and similar to the outcomes measured in their own organizations. Likewise, interviews and reviewed documents demonstrate that the approach to encouraging and measuring Equity, Diversity, and Inclusion (EDI) outcomes is matching practices being applied by other not-for-profit and research institutes. However, interviewees expressed a degree of dissatisfaction with the overall robustness of performance measurement in the sector as a whole. The general sense is that current methods and approaches are not capturing the full value of collaborations and impacts achieved through brain-related research and its translation to higher-level impacts. However, key informants had limited solutions to offer.

The current approach to performance measurement taken by the Program and organizations operating in the ecosystem is that knowledge creation and its translation into new knowledge is what creates and sustains the relationships between the multiple stakeholders in the research ecosystem. Indeed, this is consistent with the Program's current Logic Model. While dissatisfied with their respective organizations' current set of performance measures, three of the interviewees indicated that their current approach to developing new measures involves broad-based consultations with participants in the ecosystem: researchers, donors, funders, clinicians, patients, and so on. Just as research priorities for these organizations are established through these types of consultations, it appears there is a growing acknowledgement that partners and stakeholders who are involved in setting priorities should also be engaged in establishing performance measures.

This approach has the potential to capture and reflect the values and interests of those who are consulted, from patients to researchers to funders. In this way, it may be possible to capture performance and impact information that is relevant to the decision-making needs of the different groups involved in the consultation. This further reflects the nature of working in an ecosystem composed of different networks of individuals and institutions that are creating and disseminating knowledge in ways that are not necessarily being captured by current approaches to performance measurement.

7.0 Conclusion

The BCF's matching funding model and focus on supporting "high-risk, high-reward" basic research have produced research that is recognized for its excellence in the brain-related research ecosystem. Furthermore, this research addresses the research priorities of its funding partners and donors, as well as the funding priorities of the GoC. This is especially important as the financial burden of disease associated with brain and neurological health conditions reaches into the tens of billions of dollars annually, and is only increasing over time.

Furthermore, funding partners and donors have indicated that collaboration with the BCF has helped them support quality research that they may not have otherwise pursued, especially funding partners unfamiliar with brain-related research who lacked scientific and operational expertise. On top of this notable accomplishment, the BCF has met or exceeded most of its performance targets related to capacity building, demonstrating that the Program is contributing to the development of researchers in the ecosystem. Finally, some research that is being funded through BCF-funded partnerships is starting to be translated into intellectual property, demonstrating that intermediate outcomes are being achieved.

As noted in key informant interviews, the Program has developed a positive and constructive relationship with the BCF resulting in a well-managed contribution agreement. Given the maturity of the BCF, a review of the terms and conditions of the contribution agreement between the BCF and the Program is suggested to determine how to reduce process irritants, which will further support success, as well as measure and communicate impact more effectively in support of decision-making. Expanding the BCF's partnership reach internationally could result in more opportunities to fund research that increases capacity for brain research in Canada, promote the access of Canadian researchers to research platforms in other countries, and share knowledge generated by Canadian-produced research more broadly. As such, supporting international collaborations of BCF-funded researchers could both increase the expertise available to Canadian brain research efforts and broaden the dissemination and use of knowledge generated by the BCF funding partnerships at the international level.

8.0 Recommendations

The Canada Brain Research Fund Program's 1:1 matching funding model has added value to the brain research ecosystem, and has provided support for addressing the financial burden of brain and neurological diseases in Canada. The following recommendations focus on areas that could support continuous improvement in this area:

Recommendation 1: Where possible, consider any modifications to program parameters or their interpretation in order to enable greater opportunities for collaboration and potential international research partnerships.

The current terms and conditions of the Program, as well as the contribution agreement have presented challenges to creating international partnerships. This is an important matter to resolve as forging these international partnerships has the potential to increase capacity for brain research in Canada, promote researcher access to research platforms, and to share knowledge generated by Canadian-produced research more broadly.

Recommendation 2: To increase the likelihood of outcomes achievement and to measure and communicate the Program's impact for Canadians and support decision makers more effectively, consider consulting with partners, researchers and institutions in the brain-related research ecosystem to identify optimal approaches to performance measurement and knowledge translation that recognize:

1. a. the nature of a networked research ecosystem; and
2. b. the complex nature of multi-disciplinary research.

Throughout the brain and neurological research ecosystem, challenges have been identified in capturing performance data that is meaningful to funders and administrators. As well, optimal approaches for translating research knowledge to achieve intermediate and ultimate impacts are not generally agreed upon, nor applied throughout the ecosystem. Notably, there is a lack of consensus as to how to address these challenges.

9.0 Management Response and Action Plan

Evaluation of the Canada Brain Research Fund Program 2016-17 to 2020-21

Recommendations	Response	Action plan	Deliverables	Expected completion date	Accountability	Resources
Recommendation as stated in the evaluation report	Identify whether program management agrees, agrees with conditions, or disagrees with the recommendation, and why	Identify what action(s) program management will take to address the recommendation	Identify key deliverables	Identify timeline for implementation of each deliverable	Identify Senior Management and Executive (DG and ADM level) accountable for the implementation of each deliverable	Describe the human and/or financial resources required to complete the recommendation, including the source of resources (additional vs. existing budget)
Optimizing the funding agreement Where possible, consider any modifications to the program parameters or their interpretation to enable greater opportunities for collaboration and potential international research partnerships.	Health Canada program management agrees with the recommendation.	December 2021 - March 2022 – Building on previous engagement with Health Portfolio experts, the program will consult with Treasury Board Secretariat and assess how current Program parameters support the recipient's participation in international research partnerships and collaboration. December 2021 - March 2022 – The Program will engage relevant functional and subject matter areas within the Health Portfolio to validate potential provisions to be included in the current funding agreement, which is set to be extended until 2024-25.	Funding agreement reflects implementation parameters that allow international research partnerships.	March 31, 2022	Director General, Health Programs and Strategic Initiatives	This recommendation will be completed using existing SPB human and financial resources.
Updating approaches to Performance Measurement and advancing Knowledge Translation To increase the likelihood of outcomes achievement and to more effectively measure and communicate the Program's impact for Canadians and support decision-makers, consider consulting with partners, researchers and institutions in the brain-related research ecosystem to identify optimal approaches to performance measurement and knowledge translation that recognize: the nature of a networked research ecosystem; and, the complex nature of multi-disciplinary research.	Health Canada program management agrees with the recommendation	December 2021 - March 2022 – The Program will engage relevant federal stakeholders to determine the best approach for increasing the program and BCF's focus on and support for knowledge translation (KT). January - March 2022 – Using information gathered through stakeholder engagement, the Program will assess and introduce appropriate Program measures to modify/expand KT efforts. April - November 2022 – In collaboration with BCF, engage relevant subject matter experts (i.e., PM and KT research sector, broader neuroscience sector, health portfolio) to inform strategies for PM and KT suitable for adoption over the final three years of the program (from 2022-23 until 2024-25). 2022-23– For improved measurement and communication of impact, the Program will collaborate with BCF in updating the program's Performance Measurement Strategy to include approaches identified through stakeholder engagement.	Funding agreement which includes parameters to increase the Program's focus on and support for KT Summary of engagement findings on PM and KT with guiding recommendations Revisions to Performance Measurement Strategy	March 31, 2022 November 30, 2022 May 30, 2023	Director General, Health Programs and Strategic Initiatives	This recommendation will be completed using existing SPB human and financial resources.

Appendix 1: Comparative analysis chart

	Brain Canada Foundation	Institute of Neurosciences, Mental Health and Addiction (INMHA – CIHR)	Institute of Aging (IA – CIHR)	Ontario Brain Institute	Michael Smith Foundation for Health Research	Hotchkiss Brain Institute
Funding Source	Public and Private Foundations (7%) Corporations (3%) Private Donors (1%) Health Charities (33%) Provincial Agencies (8%) Research Network (1%) Institutions (36%) Other Organizations (11%)	On average, investments in research relating to INMHA's mandate represented 24% of total CIHR investments over the period from 2000-01 to 2014-15. Total investments in 2014-15 in this area were $229.6M.	Over the past 17 years, the average annual percentage of CIHR investment in IA mandate was 11% of CIHR's total annual investment.	Provincially funded by the Government of Ontario: $40 million in 2018-19 $40 million in 2019-20 In 2020/2021, $40 million (with only $24 million leveraged) in 2020-21 $40 million in 2021-22 $40 million in 2022-23	Funded by the Government of British Columbia: $22M in revenue in 2019-20 $24M in revenue in 2020-21	In 2017-18, the Institute received more than $47M in total research funding.
Research Priorities	Brain Cancer Knowledge Translation Neurodegenerative Neurodevelopmental Neuropsychiatric and Mental Health Pain and Migraine Seizure Sensory System Stroke/Injury	Mental health Neurological health Spinal cord Vision, hearing and cognitive functioning Three primary research priorities: Opioids and Substance Use Cannabis Post-Traumatic Stress Emerging priorities: Concussion Mental health services	Health and wellness along the life-long trajectory of aging Health challenges of older individuals Specific research areas include: Chronic conditions Dementia Healthcare and services Information on late-life care and decisions Research priorities: Cognitive impairment and dementia Home care Preventative lifestyle measures Mobility challenges Biology of aging mechanisms	Testing of biomarkers in community care settings Advanced analytics for disease modeling and diagnostics Quality improvement processes for health care New treatments Implementing a world-class informatics platform Scale up small companies Attract and develop new management talent and increase employment in the neurotech cluster Patient research priorities addressed in research Health system planning using research and administrative data Data-driven decision making and policy development	To inform and respond to emerging population health and health system issues. No specific brain or neurological focus.	Research goal will be reached with the introduction of a "NeuroDiscovery Framework", which aligns research within three themes of Brain and Behaviour (epilepsy, mental health and stress, neurodevelopment), Neural Injury and Repair (multiple sclerosis, spinal cord and nerve injury, and concussion research) and Healthy Brain Aging (dementia and cognitive disorders, movement disorders, and stroke).
Scientific Direction (Mandate)	To understand the brain, in health and illness, to improve lives and achieve societal impact by: Increasing the scale and scope of funding to accelerate the pace of Canadian brain research Creating a collective commitment to brain research across the public, private and voluntary sectors Delivering transformative, original and outstanding research programs	To support research that: Enhances knowledge of the brain Reduces the burden of brain illness through prevention strategies, screening, diagnosis, treatment, support system and palliation Improves the understanding of human thought and emotion, behaviour, sensation, perception, learning, and memory	To support research, to promote healthy aging and to address causes, prevention, screening, diagnosis, treatment, support systems, and palliation for a wide range of conditions associated with aging." Its goal is to improve the quality of life and health of older Canadians by understanding and addressing or preventing the consequences of a wide range of factors associated with aging.	Collaborative approach to research aims to: Enhance the neuroscience research system, Grow the neuro-technology cluster, Improve brain health for Ontarians	Foster talent development so BC can develop, attract, and retain the very best health researchers Support research that informs the provincial health system, specific system priorities, and innovation Build capacity in health research uptake; and optimize provincial health research investment through partnerships	Mission is to inspire discovery and apply knowledge towards innovative solutions for neurological and mental health disorders. The primary strategic research goal of the HBI is to achieve internationally recognized key discoveries and transformative clinical research in the neurosciences and mental health.
Knowledge Management/ Knowledge Translation (KM/KT) Activities (Checklist)	No specific KT Program, although it has funded projects with a KT focus.	No specific KT Program	No specific KT Program	The GEEK (Growing Expertise in Evaluation and Knowledge Translation) program provides funding, evaluation expertise, and support to community-led programs and services for people living with brain disorders. GEEK supports the sustainability, scaling up, and spread of these programs, to improve the quality and quantity of evidence-based care in the community.	Fully developed KT strategy, paired with an evaluation framework, and based on an understanding of complex systems developed to increase the effectiveness and efficiency of research.	Has a Knowledge Engagement business line, with a focus on Indigenous Research Support. This business line is offered to support community organizations to help establish and find research partners, support collaborations, build capacity, and mobilize knowledge.

Appendix 2: Impacts of two-year funding for contribution agreements

The first two-year funding agreements were implemented in 2019.

Figure 1: Text description

This image depicts the various steps of the contribution agreements for project funding over the course of 6 years. At the bottom of the image, there is a timeline divided into 6-month segments, with numbers of months from 0 to 72. Under this timeline are the various steps of the funding process, including GoC Funding Announcement, GoC Funds Available, Research Program funds commence, Accountability/funding milestone (3 times), and Project conclusion. At the top of the image, above the timeline is a series of horizontal coloured bars covering various periods on the timeline, including, from right to llest a light purple bar for "Contribution Agreement approval process" (12 months), a sky blue bar for "Research program partnership; design; project review/approval" (12 months), a pale green bar for "Research Project Duration (3-year example)", and a green bar for "Optional 1-year extension". Above the 24-month and 60-month marks of the timeline is the text "Agreement disbursement deadline".

Figure 2: Text description

The image is comprised of a series of multi-coloured vertical bars that depict the length of research projects as they relate to the year of the funding competition.

• For 2016, according to the number of funding competitions launched that year, the length of research projects was, in descending order, 3 years, 4 years, 2 years, 5 years, 1 year, 6 years, and 7 years.
• For 2017, according to the number of funding competitions launched that year, the length of research projects was, in descending order, 2 years, 2 years, 3 years, 1 year, and 4 years.
• For 2018, according to the number of funding competitions launched that year, the length of research projects was, in descending order, 3 years, 2 years, 1 year, 4 years, and 5 years.
• For 2019, according to the number of funding competitions launched that year, the length of research projects was, in descending order, 3 years, 2 years, 4 years, and 1 year.
• For 2020, according to the number of funding competitions launched that year, the length of research projects was, in descending order, 2 years, 1 year, 3 years, and 4 years.
• For 2021, according to the number of funding competitions launched that year, the length of research projects was, in descending order, 2 years, 3 years, 1 year, and 4 years.

Appendix 3: Evaluation methods and limitations

The evaluation used multiple lines of evidence, such as a literature and document review, performance data, and key informant interviews to ensure triangulation of findings.

Key Informant Interviews

OAE interviewed 39 key informants, including:

• Academic and Research Institutions (8)
• BCF Staff and Executives (7)
• Federally-funded research institutes (6)
• NGO funding partners (5)
• HC Program representatives (5)
• Private funding partners (5)
• Provincial research institutes (3).

Document Review

OAE reviewed over 75 internal and public documents related to program delivery, including, but not limited to:

• BCF Annual Reports
• Briefing and Planning Documents
• Proposals and other foundational documents.

Comparative Analysis

A comparison of the BCF to seven other brain-related research institutes (six Canadian and one international) was conducted based on eight factors. The analysis was based on a review of publicly available program documents and three key informant interviews.

Administrative and Performance Data Review

A review of administrative and performance data was completed to assess progress towards stated objectives. Administrative data included program delivery information from the BCF databases, and performance data included a review of case studies and PIP data covering the evaluation period.

Literature Review

Two reviews of relevant peer-reviewed and grey literature published between 2014-15 and 2020-21 were conducted with support from Health Canada's Health Library Information Branch (HLIB), in addition to open-source searches. A total of 41 articles were selected for inclusion.

As with many evaluations, there were some limitations encountered during the implementation of the selected methods that may have had implications for the validity and reliability of evaluation findings and conclusions. The following table provides a summary of the limitations, impact, and mitigation strategies to ensure the findings could be used with confidence to guide program planning and decision making.

Table 6: Summary of limitations, impact, and mitigation strategies

Limitation	Impact	Mitigation Strategy
The sample size of key informants was relatively small, considering it was divided across several groups of stakeholders.	Key informant views may not be equally representative across groups due to the different distribution of key informants in each of the groups.	Information from key informant interviews was triangulated with key documents containing performance data.
Key informant interviews are retrospective in nature and are sometimes limited to specific points in time (i.e., the duration of a specific project).	Interviews tend to provide recent perspective on past events. In some cases, key informants had been involved with the Program for a limited amount of time, such as new hires.	Triangulation with other lines of evidence substantiated or provided further information on data captured in interviews. Document review provided corporate knowledge.
A few of the staff key informants either had left their previous postings or had recently joined, which affected their level of corporate knowledge to answer interview questions.	In a few cases, key informants did not have the same corporate knowledge and were not able to respond to some of the questions in the interview guide.	Key informants with corporate knowledge were included to balance the information received adequately. Triangulation with other lines of evidence such as the document review also provided corporate knowledge information.