Evaluation of PMRA's Activities in Support of the Minor Use Pesticide Program 2013-14 to 2019-20

Prepared by
Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

March 2021

Table of Contents

• List of Acronyms
• Executive Summary
• Management Response and Action Plan
• 1. Evaluation purpose
• 2. Program profile
• 3. How the Minor Use Pesticides Program (MUPP) works
• 4. Findings
  • 4.1. Evidence-based decisions in review and approval processes
  • 4.2. MUPP is a Valued Partner in the Agricultural Sector
    • 4.2.1. Scientific Expertise
    • 4.2.2. Program Governance and Delivery through Collaboration
• 5. Challenges and Areas of Improvement
  • 5.1. Re-examining performance targets commensurate with complexities and level of effort, and workload management
  • 5.2. Transparency, communication and other external challenges
    • 5.2.1. Communication and Transparency of the Review Process
    • 5.2.2. Re-evaluations and Other External Challenges
• 6. Program Resources
• 7. Conclusions
• 8. Recommendations
• Appendix 1 – MUPP Logic Model
• Appendix 2 – Performance Timelines and Services Standards for MUPP Submissions
• Appendix 3 – Scope, Evaluation Questions and Data Collection Methods
• Appendix 4 – MUPP Process Submission Flowchart
• Appendix 5 – Performance Data of PMRA's Activities in Support of the Minor Use Pesticides Program
• Endnotes

List of Acronyms

AAFC

Agriculture and Agri-food Canada

ADM

Assistant Deputy Minister

ATIP

Access to Information and Privacy

DM

Deputy Minister

EPA

Environmental Protection Agency

FPT

Federal/Provincial/Territorial

HC

Health Canada

IR-4

Interregional Research Project No. 4

MOSP

Management of Submission Policy

MOU

Memorandum of Understanding

MRL

Maximum Residue Limit

MUPP

Minor Use Pesticide Program

PMRA

Pest Management Regulatory Agency

PMUC

Provincial Minor Use Coordinators

PT

Provincial and Territorial

SPN

Science Policy Note

U.S.

United States

Executive Summary

Program Profile

The Minor Use Pesticides Program (MUPP or "The Program") is a joint initiative between Agriculture and Agri-food Canada (AAFC) and Health Canada's Pest Management Regulatory Agency (PMRA) that has been in place since 2003. It has been funded every five years through a Memorandum of Understanding (MOU), whereby $4 million is transferred annually to PMRA.

A "minor use" pesticide refers to the use of crop-protection treatments (like herbicides, insecticides, fungicides, and nematicides) on minor crops that are of low acreage, high-value, or, where pest control is only needed on a small portion of the overall crop acreage. include fruits, vegetables, herbs and spices, landscape ornamentals and specialty crops. They tend to be grown on smaller areas of land compared to large crops like wheat, soybean, and corn.^Footnote 1 The projected sales of some pest control products may be so low that manufacturers may conclude that the sales potential is not sufficient to justify the investment required to register a particular use in Canada.

The Program provides benefits to Canadian producers by making minor use pesticide products more readily available, as well as improving Canadian producers' competitiveness domestically and internationally.

PMRA reviews pre-submission requests from AAFC and the provinces to ensure applications meet the necessary data requirements. With support from manufacturers, AAFC and provinces develop application packages containing any required data for submission to PMRA. PMRA conducts scientific regulatory reviews and delivers a proposed regulatory decision. If the use has acceptable risk and value, the registrant can make an application to add the minor use(s) to the pesticide product label.^Footnote 2

What we found

Overall, the evaluation found that PMRA has successfully met its MOU objectives and has exceeded performance targets, contributing to an increase in the availability of minor use pesticides to growers. However, the number of submissions received per year far exceeds the performance target, creating an increasing number of applications in queue to be completed.  Considering that funding has been static for the past 17 years, any increase in program activities may have implications for program resources.

As a whole, stakeholders perceived the MUPP to be of great value, with a good reputation as a world leader in the minor use pesticides field. PMRA's submission reviews and regulatory decisions on approving minor uses of pesticides are considered to be evidence-based and a credible source of scientific expertise and knowledge.

While the findings showed that PMRA is meeting its MOU commitments, some challenges were identified:

• The need to re-examine performance targets to better reflect the time and level of effort required to assess incoming submissions, which have become increasingly complex due to global, environmental and agricultural changes (e.g., the emergence of unique crops and new technologies);
• A perceived lack of transparency in the way PMRA communicates to stakeholders regarding the status of the review and approval processes; and
• An unpredictable workload, since applications are not received based on a set schedule, which affects the management of internal resources.

The evaluation findings discussed in this report led to the following recommendations:

Recommendation 1: To facilitate decision making as the level of complexity increases due to new technologies, new crops, multiple submissions, as well as joint reviews, PMRA should determine the length of time and level of effort required to complete and process submissions received.

The MOU's budget has not increased since the inception of the Program seventeen years ago, despite the evolution of the minor use pesticides landscape. Submissions to PMRA have become more complex, sometimes requiring additional data or resulting in a longer time to complete health risk assessments. This often occurs with submissions for crops newly authorized to be grown in Canada (e.g., wasabi, rice) and new methods or technologies to apply minor use pesticides (e.g., use of drones, bulb dips).

In this context, current performance targets do not provide the Program with a clear understanding of how long it takes and the level of effort needed to review each submission to manage the distribution of workload each year. This is especially important as applications are not received based on a set schedule and the number of submissions received has been increasing, leading to a growing number of applications in queue to be completed each year. Having a clearer picture of the level of effort would help the Program to better understand the level of resources required to complete submissions, manage workload and inform possible revisions to the MOU in 2023.

Recommendation 2: Increase communication and awareness both internally and externally to address a perceived lack of transparency with respect to PMRA's processes for reviewing submissions, and to clarify and manage stakeholder expectations.

Both internal and external key informants (AAFC, PTs, registrants, and growers) mentioned timelines and transparency as ongoing issues, which are increasing stakeholder frustration.  Although legal restrictions limit PMRA's ability to discuss certain details about upcoming regulatory decisions (e.g., re-evaluations), clearer communication with internal and external stakeholders may help to manage expectations and improve overall understanding of the internal processes and regulatory limitations.

Management Response and Action Plan

1. Evaluation Purpose

The purpose of the evaluation was to assess the effectiveness and efficiency of PMRA's activities in support of the Minor Use Pesticide Program (MUPP) for the period from April 2013-14 to March 2019-2020.

For details on the evaluation scope, approach, and design, including the MUPP logic model, evaluation questions, and a description of the data collection and analysis methods, limitations, and mitigation strategies, please consult Appendix 3.

2. Program Profile

A "minor use" pesticide refers to the use of crop-protection treatments (like herbicides, insecticides, fungicides, and nematicides) on minor crops that are of low acreage, high-value, or, where pest control is only needed on a small portion of the overall crop acreage. include fruits, vegetables, herbs and spices, landscape ornamentals and specialty crops. They tend to be grown on smaller areas of land compared to large crops like wheat, soybean, and corn.^Footnote 3 The projected sales of some pest control products may be so low that manufacturers may conclude that the sales potential is not sufficient to justify the investment required to register a particular use in Canada.^Footnote 4

The Minor Use Pesticides Program (MUPP or "The Program") was launched in 2003 as a joint initiative between Agriculture and Agri-food Canada (AAFC) and Health Canada's Pest Management Regulatory Agency (PMRA). This initiative aims to provide Canadian producers with access to new pest-management technologies and improving their competitiveness domestically and internationally. As part of a larger AAFC initiative, PMRA receives $4 million annually to support the MUPP in making minor use pesticide products more readily available to growers.

The expected outcomes of the Program are the following:

• New minor uses of pesticides are available to growers;
• Stakeholders implement strategies and use tools to manage changes associated with regulatory modernization and crop protection; and
• The agriculture, agri-food and agri-based products sector is able to adapt to a changing regulatory and crop protection environment with scientifically evaluated tools.

The MUPP logic model is available in Appendix 1 for a visual representation of the links between activities, outputs, and results.

3. How the Minor Use Pesticides Program (MUPP) works

The Program is comprised of several activities delivered by the Pest Management Centre (AAFC) and the PMRA (Health Canada). AAFC, in collaboration with stakeholders, including grower representatives, minor use registrants, Provincial Minor Use Coordinators, and international bodies, such as the U.S. Interregional Research Project No. 4 (IR-4) and the Environmental Protection Agency (EPA), identifies pest control needs (international or domestic), for minor uses of pesticides, and facilitates an annual prioritization process.

If AAFC identifies similar priorities as those in the United States (U.S.), the Pest Management Centre and IR-4 will collaborate by conducting joint projects or workshares. In the case of joint projects, PMRA works with its U.S. counterpart, the EPA, to review these projects when they are jointly submitted to both Agencies.

Depending on circumstances, there are three steps in PMRA's review process that were included in this evaluation:

• PMRA reviews pre-submission requests from AAFC (including joint projects between AAFC and US IR-4) and the provinces noting the required data requirements. With support from manufacturers, AAFC and the provinces develop application packages containing any required data for submission to PMRA (classified as a D.3.1 submission);
• Once AAFC or the provinces provide all the information required, PMRA then conducts a scientific assessment of the health and environmental risks, as well as a value assessment of the pesticide products included in the submission, resulting in a proposed regulatory decision (classified as a D.3.2 submission); and
• If the risk and value assessments are deemed to be acceptable, the registrant can then submit a formal application to legally add the minor uses to the pesticide product label (classified as a C.6.3 submission).

Only the formal application to add the minor use to the label (C.6.3) is subject to PMRA's Management of Submission Policy (MOSP), with specific service standards, whereas the pre-submission check (D.3.1) and submission review (for risk and value) (D.3.2) use annual targets, as specified in the MOU, for number of submissions completed and new uses registered.  Appendix 2 provides a table of the complete set of performance timelines and service standards for MUPP submissions.

It should be noted that Health Canada's new cost-recovery regime includes a mechanism to ensure that the industry's share of the cost of registering pesticides remains consistent each year. Fees are therefore adjusted annually by 2% to account for inflation and other cost increases. Under the Service Fees Act, Health Canada also plans to review the cost recovery regulations every three to five years.^Footnote 5

In the context of the MOSP, once the fee remission policy comes into effect in 2021, registrants will be reimbursed part of their registration fees if the processing of these submissions does not meet the set performance timelines. This would then impact PMRA's cost-recovery revenues.^Footnote 6

4. Findings

4.1. Evidence-based decisions in review and approval processes

PMRA is meeting the objectives and commitments stated in the MOU. Over the period covered by the evaluation, PMRA generally met or exceeded the performance targets set in the MOU, thus increasing the availability of minor use pesticide products to growers.

PMRA is successfully contributing to an increase in the availability of minor use products to growers. These findings are in line with a recent evaluation conducted by AAFC in 2017 that also found that the MUPP filled an ongoing need for grower access to pesticides by increasing the availability of new pesticides for minor uses.^Footnote 7

Figure 1 provides data on the number of applications for which a scientific and value assessment is carried out (D.3.2) that PMRA received and completed between 2013 and 2019. The figure shows that PMRA received an average of 110 applications for review per year from both AAFC and the provinces. On average PMRA completed 93 applications, exceeding the yearly targets of 75 and 50 set out in the 2013 and 2018 MOUs, respectively. Similarly, the Program completed an average of 12 joint reviews per year, almost twice the set target of seven.

Overall, over the period covered by the evaluation, PMRA reviewed and made 649 regulatory decisions. PMRA also processed 427 formal applications to legally add the minor uses to the pesticide product label (C.6.3), initiated by registrants, an average of 61 applications per year, adding 3,835 new minor uses to labels. On average, these numbers represented 548 registrations of new minor uses per year, which is well over the MOU target of 200 for new minor uses registered each year.  Furthermore, based on MOSP service standards, 90% of applications submitted by registrants to receive a signed certificate to legally add the minor uses to a pesticide product label (C.6.3), are completed on time.

Key informants noted that the higher number of applications provided to PMRA for review, relative to the target identified in the MOU, has created workload management challenges and was one factor contributing to the growing number of applications in queue to be completed.^Footnote 8

Figure 1 - Text Description

The D.3.2 received column (blue) shows number of D.3.2 MUPP submissions received annually between 2013 and 2019. The number of D.3.2 completed annually were; 93, 121, 130, 104, 97, 134 and 92 respectively.

The D.3.2 completed column (orange) shows number of D.3.2 MUPP submissions received annually between 2013 and 2019. The number of D.3.2 completed annually were; 90, 106, 114, 100, 60, 76 and 103 respectively.

The target D.3.2 completed column (grey) established the benchmark as to the minimum of D.3.2 submissions to be completed annually. This number was set as 75 between 2013 and 2017 and decreased to 50 in 2018 and 2019.

4.2. MUPP is a Valued Partner in the Agricultural Sector

The Program is well received and highly valued in the Minor Use Pesticides community. Stakeholders expressed confidence in PMRA's evidence-based scientific expertise, both domestically and internationally; not only for its ability to introduce many new products to the sector, but also for increasing growers' competitiveness in the international field.

4.2.1. Scientific Expertise

A significant portion of the MUPP's activities involve the use of scientific expertise to assess submissions to expand pesticide labels. In this respect, the Program is highly valued for its scientific expertise both domestically and internationally. All stakeholders engaged in the evaluation noted that the Program is operating well, and is well recognized for its ability to increase access to minor use pesticides in the agri-food and agricultural sectors. It should be noted that, while most stakeholders expressed the desire to have more transparency from PMRA around the review process, some also recognized that, when contacted directly with specific inquiries, PMRA's responsiveness efforts were commendable (See Section 5.2 on transparency, communication, and other external challenges).

Interested parties such as AAFC, provincial coordinators, the U.S. EPA, growers, and registrants also regard the Program as a world leader because of its evidence-based regulatory decisions that include a high level of collaboration and engagement. Participants stated that PMRA staff are well respected and are seen, domestically and internationally, as a credible source of scientific expertise and knowledge.

One key informant noted that the MUPP is a testing ground for new technologies and new crops (e.g., use of drones to apply pesticides, novel crops in Canada like wasabi or rice). In some instances, minor use submissions give PMRA an early signal about where to build capacity for new technological areas or where new or specialty crops are emerging.

4.2.2. Program Governance and Delivery through Collaboration

The MUPP has established good program governance through its internal and external collaborative committees and has been able to better adapt to the evolving landscape of the minor use pesticides industry. In this context, PMRA has also supported various measures to improve program delivery, such as with the bundling of submissions, conducting joint reviews between Canada and the U.S., harmonizing data requirements, aligning residue trials, and revising guidelines and directives (i.e., the Value Guidance Directive) to further reduce data requirements.

The MUPP fosters successful collaborations between the Government of Canada, provincial representatives, registrants, and growers who support work in the pest management cycle to provide Canadian producers access to new pest-management technologies and improve their competitiveness domestically and internationally. Over the years, the Program has adapted to the changing environment with nimbleness, responsiveness, and continuous course corrections (e.g., revising guidelines to reduce data requirements, working on multiple submissions and joint reviews as discussed later in more detail).

According to both internal and external stakeholders, the strength of the Program is that it is driven from the ground up, with registrants and growers coming together to identify their needs during an annual prioritization meeting which is facilitated by AAFC. The prioritization meeting has been touted as an important aspect of the Program whereby different groups and interests are brought to the table to discuss the most pressing needs, issues, and challenges. PMRA's participation is also viewed as an effective way for the Agency to gain early intelligence on some of the most pressing problems with pests, invasive species, and diseases, and potential solutions. It is also an opportunity for PMRA to flag to registrants the uses where emergency registrations require long-term solutions, or chemicals that may be under re-evaluation, the outcome of which can impact regulatory conditions.

The partnership between PMRA and AAFC ensures good program governance through various internal committees, such as a Joint Management Committee made up of Assistant Deputy Ministers and Directors General who meet yearly, and a program level working group that has bi-weekly meetings. These internal committees provide a regular forum for exchanges on progress and work activities that assist with the monitoring and management of the MUPP.

Other collaborative mechanisms that are part of the MUPP, such as the Federal/ Provincial/Territorial (FPT) Minor Use Sub-committee, are beneficial as they allow all parties to stay informed on national issues related to minor uses for pesticides. For instance, some provincial and territorial partners mentioned that regular telephone calls every three months provided additional information on a range of questions related to such issues as emergency registrations, processes, and new regulatory needs. Along the same lines, PMRA fosters external stakeholder relationships by attending a variety of stakeholder meetings throughout the year, such as the Prairie Pesticide Minor Use Consortium and the Canadian Horticulture Council. Upon request, during such meetings, the Agency may provide updates on the work completed, discuss any upcoming issues, and provide details on specific submissions. Finally, PMRA and PMC staff also do site visits to farms and greenhouses to understand better the needs of the agricultural sector.

Based on these collaborative frameworks, PMRA has also been able to adjust some of its processes to address stakeholders' feedback and make some course corrections. For instance, with the publication of the "Value Guidelines", PMRA refined and reduced the value data requirements for projects selected at the AAFC Priority Setting Workshop. In other words, the purpose of the guideline was to potentially shorten the time required for data generation by AAFC, Provincial Minor Use Coordinators, and registrants, reducing overall submission costs. Also with this guideline, in cases where a submission is based on the highest priority level, as selected by the Canadian Minor Use Priority Setting Workshop, PMRA recognizes value based on the prioritization process and does not require trial data to be submitted for regulatory review. In most cases, as mentioned by PMRA's staff, this can reduce the effort during the review. Similarly, also to save some time during the review process, PMRA has bundled multiple minor use requests into one submission following a request from provinces to add multiple crops to the same pesticide label.

Internationallythe MUPP works collaboratively with the U.S. IR-4 and EPA. For instance, as mentioned earlier, once national priorities are identified, the PMC and PMRA collaborate with IR-4 and EPA to choose projects that meet minor use pesticide needs both in Canada and the U.S. PMRA and EPA conduct joint reviews with the aim of both harmonizing and aligning residue trial requirements and Maximum Residue Limits (MRLs) for MUPs in Canada and the U.S., as well as exchanging best practices. Overall, joint reviews have been able to save time and effort on generating data, which has reduced up to 63%^Footnote 9 the total number of field trials required to support registration in both countries.^Footnote 10 Moreover, this collaboration also aims to address challenges related to complex submissions.

Finally, internal and external key informants noted that the relationship between AAFC and PMRA seems, for the most part, to be working well. Some concerns were voiced by internal key informants on whether the MOU performance targets are still meeting the needs of all parties. Specific concerns focused on the increase in the numbers of submissions PMRA receives for review, which exceed the current performance targets identified in the MOU, resulting in an increased workload at PMRA, accompanied by longer review timelines and transparency issues. These concerns will be further discussed in the following section.

5. Challenges and Areas of Improvement

5.1. Re-examining performance targets commensurate with complexities and level of effort, and workload management

While PMRA has been successful in meeting its targets, the majority of internal and external stakeholders seemed to suggest that the targets are not commensurate with current realities, both in terms of the number of submissions that are received by PMRA, and in the amount of time it can take for the review and approval processes.

As some internal and external key informants explained, submissions may vary in their level of complexity, thus affecting the length of time for review and the level of effort required, but there is no system currently in place that assesses and categorizes submissions based on these factors. For instance, while bio-pesticides can take less time to review, as they do not often require the same type or extent of scientific data, new crops or technologies can be more complex and require increased efforts in the health risk assessments to fulfill regulatory requirements. In the case of new crops or with new technologies (e.g., drones), PMRA has had to consult more often with stakeholders and may require additional data in order to better understand the type of analysis required, increasing the time it may take to process and review submissions. Introducing new submission categories to reflect the level of review effort may help to more accurately determine the amount of time needed for the actual review and approval process.

Some key informants suggested that the current targets are not an accurate representation of level of effort and of the work required to review and process submissions. In this regard, some also suggested that, in order to have more meaningful measures, a combination of targets and timelines, similar to those used for the MOSP (PMRA's performance service standards), would be more appropriate. This would also help PMRA be more transparent with its stakeholders in managing expectations on the length of time it takes submissions to be reviewed and approved.

Similarly, while PMRA is successfully meeting its targets, most internal stakeholders seemed to suggest that the targets are not commensurate with current realities, both in terms of the number of submissions received by PMRA, and the amount of time required for the review and approval processes.

In fact, not only does PMRA receive more submissions to review than the target established in the MOU, data collected showed increasingly longer timeframes to process submissions.

For instance, as mentioned earlier, PMRA uses internal service standards to track its submissions using guidelines similar to those used in the MOSP. The data showed that there is an increasing trend in the length of time it takes to review these submissions when compared to previous years. For example, in 2013-14, it took an average of 264 days to conduct a scientific review of a submission, which is below the performance target of 285 days. However, in 2019-20, the average time to conduct a scientific review of a submission took 384 days to complete, or about 33% above the target number of days (please see Appendix 5, Table 6).

Along the same lines, internal key informants also stated that workloads are difficult to predict each year, due to large fluctuations in the timing of submissions sent to PMRA. Similarly, the higher number of applications received for review was also flagged by internal key informants as one of the factors that has created challenges in terms of workload planning and management, and as one of the factors contributing to the growing number of applications in queue to be completed.^Footnote 11

To this extent, the unpredictability of the workload, coupled with the fact that the number of submissions received often exceeds the established target in the MOU, are two notable factors that may have contributed to the increasing number of submissions in queue to be completed, as well as the increasing timelines in the review of submissions. Of note, recent records of decisions from Joint Management Committee meetings held in 2020 suggest that PMRA and AAFC had already flagged similar issues around the increasing complexities of incoming submissions, as well as the need to re-examine the costing analysis of the MUPP to inform the next round of funding and possible revisions to the MOU for 2023.

As the Program budget has remained at $4 million per year since its inception, PMRA should consider revisiting the current performance targets to reflect the different categories of submissions received. Factors such as the various complexities, and the level of effort and time required to complete specific reviews might be considered during work planning stages to reflect more accurately the Program's resource needs.

5.2. Transparency, communication and other external challenges

The perceived lack of transparency of the review process, along with the length of time for submissions to be reviewed and approved was a frequent source of frustration, as voiced by both internal and external key informants. Additional emerging challenges mentioned were related to the evolving globalization of the industry, such as re-evaluations, emergency registrations, new technologies, and the use of new agriculture tools (e.g., drones) or finding new minor uses for pesticides to combat invasive species.

5.2.1. Communication and Transparency of the Review Process

It was also noted by most external participants, as well as some internal key informants, that there were gaps in communication provided by PMRA regarding the status of the review and approval processes. Some key informants referred to internal processes as a "black box" and expressed a desire to receive more information on their application status, as they were not clear as to why review timelines were so lengthy. For example, applicants receive auto-generated emails twice during the application cycle providing them with a submission number as well as an email to let them know their submission is in review. At the same time, they receive little information on where the application is in queue and the length of time expected for the review to be completed. However, PMRA is also limited in its ability to share specific information with stakeholders in order to safeguard regulatory privacy issues and program integrity.

Despite the noted limitations, however, clearer communication with internal and external stakeholders may benefit PMRA in managing stakeholder expectations on timelines and increase awareness as to why certain submissions are not reviewed (i.e., due to re-evaluations) or why the process takes longer than anticipated.

5.2.2. Re-evaluations and Other External Challenges

Re-evaluations of pesticides are conducted every 15 years to ensure that regulated pesticides still meet scientific and health safety standards. The removal of pesticides or specific uses from the market due to these re-evaluations was noted as a growing concern by all key informants, with direct impacts to the availability of minor use pesticides. As well, re-evaluations were mentioned as affecting review timelines and increasing issues around transparency, as registrants were not made aware, due to regulatory privacy issues, of why certain submissions were not reviewed.

A few stakeholders went even as far as to say that re-evaluations undermined the intended objectives of the Program, as this process can result in certain minor uses being removed from the market, which then impacts growers' competiveness (by potentially making less pesticides available to growers). They believed the efforts spent on generating data for the re-evaluation process, reviewing it, and approving minor uses were therefore wasted when the pesticides were removed from the market. In addition, stakeholders expressed a lack of understanding of how decisions were made and stated they felt "left out" of the process due the lack of information or communication from PMRA. 

Other external challenges were linked to rising complexity in market globalization. For instance, PMRA has facilitated the alignment of MRLs with those in the U.S. Some stakeholders emphasized the need to continue to improve harmonizing MRLs to avoid non-tariff trade barriers and to continue to ensure Canada's competitiveness in global markets. Industry stakeholders felt this was a more pressing issue, as buyers could tap into foreign markets that have higher MRL limits thus, in the medium term, decreasing the demand for Canadian products.

Further challenges were noted in cases related to invasive species and fast-tracking submissions in cases of emergency, novel crops, the intensification of greenhouse use in Canada, and new technologies in agriculture, such as the use of drones. To examine these submissions, new data is often needed, notably longitudinal data, to assess the longer-term health risks related to these new uses and to learn about the production of these crops. Therefore, it can take AAFC or the provinces up to two years to develop the data required to submit a pre-submission application package for a minor use pesticide to PMRA for new invasive species. The need for this new data requires additional resources during the pre-consultation stages, which can cause further delays in processing submissions within the performance timelines.

6. Program Resources

Over the evaluation period, the Program spent approximately 89% of its budget. Underspending in the last two years was primarily due to staffing vacancies.

Table 1 presents planned and actual program expenditures for the period between 2013-14 and 2019-20. During the evaluation period, the Program lapsed approximately $2.8 million, largely attributable to difficulties in staffing positions that required scientific expertise. The budget also showed some underspending in O&M between 2015-16 and 2019-20, as some of the planned translation costs were not incurred.

It should be noted that PMRA's funding has not changed since 2003, remaining static at $4M per year. To this point, static funding has not impacted the Program's ability to successfully meet its MOU objectives. However, program resources may not be sufficient to address some of the challenges already noted in previous sections, such as managing the increasing number of submissions received each year, including the complexity of these submissions due to global, environmental and agricultural changes.

7. Conclusions

Overall, the evaluation found that PMRA has successfully met its MOU objectives and, in the context of static funding for the past 17 years, exceeded performance targets. These results have contributed to an increase in the availability of minor use pesticides to growers. Some key informants, however, noted that the number of submissions received per year exceeds the performance targets identified in the MOU, and that this could be one of the reasons for the increasing backlog in the number of submissions for review.

As a whole, stakeholders perceived the MUPP to be of great value, with a good reputation as a world leader in the minor use pesticides field. PMRA's submission reviews and regulatory decisions on approving minor uses of pesticides are considered to be evidence-based and a credible source of scientific expertise and knowledge.

While the findings showed that PMRA is meeting its MOU commitments, some challenges were identified:

• The need to re-examine performance targets to better reflect the time and level of effort required to assess incoming submissions, which have become increasingly complex due to global, environmental and agricultural changes (e.g., the emergence of unique crops and new technologies);
• A perceived lack of transparency in the way PMRA communicates to stakeholders regarding the status of the review and approval processes; and
• An unpredictable workload, since applications are not received based on a set schedule, which affects the management of internal resources.

8. Recommendations

The evaluation findings discussed in this report led to the following recommendations:

Recommendation 1: To facilitate decision making as the level of complexity increases due to new technologies, new crops, multiple submissions, as well as joint reviews, PMRA should determine the length of time and level of effort required to complete and process submissions received.

The MOU's budget has not increased since the inception of the Program seventeen years ago, despite the evolution of the minor use pesticides landscape. Submissions to PMRA have become more complex, sometimes requiring additional data or resulting in a longer time to complete health risk assessments. This often occurs with submissions for crops newly authorized to be grown in Canada (e.g., wasabi, rice) and new methods or technologies to apply minor use pesticides (e.g., use of drones, bulb dips).

In this context, current performance targets do not provide the Program with a clear understanding of how long it takes and the level of effort needed to review each submission to manage the distribution of workload each year. This is especially important as applications are not received based on a set schedule and the number of submissions received has been increasing, leading to a growing number of applications in queue to be completed each year. Having a clearer picture of the level of effort would help the Program to better understand the level of resources required to complete submissions, manage workload and inform possible revisions to the MOU in 2023.

Recommendation 2: Increase communication and awareness both internally and externally to address a perceived lack of transparency with respect to PMRA's processes for reviewing submissions, and to clarify and manage stakeholder expectations.

Both internal and external key informants (AAFC, PTs, registrants, and growers) mentioned timelines and transparency as ongoing issues, which are increasing stakeholder frustration.  Although legal restrictions limit PMRA's ability to discuss certain details about upcoming regulatory decisions (e.g., re-evaluations), clearer communication with internal and external stakeholders may help to manage expectations and improve overall understanding of the internal processes and regulatory limitations.

Figure 2 - Text Description

This figure depicts the component of the Minor Use Pesticides Program (MUPP) logic model, which is a visual illustration of the program's objectives, activities, outputs and expected outcomes.

The MUPP is a joint initiative between Agriculture and Agri-Food Canada (AAFC) and Health Canada's Pest Management Regulatory Agency (PMRA).

The objectives of the program are the following:

• The program will address the needs of Canadians growers for new minor uses of pesticides and will assist stakeholders to be sustainable and remain competitive in the market place.
• The program will provide tools that will enable stakeholders to modernize crop protection within Canada.

MUPP conducts the following activities: AAFC identifies stakeholder priorities for minor use pesticides; AAFC generates scientific data, prepares and sends regulatory submissions for new minor use pesticides; reviews and evaluates regulatory submissions of new minor uses of pesticides that are submitted by AAFC and the provinces.

These activities produce several outputs:

• An MOU between AAFC and Health Canada’s PMRA
• The documentation of stakeholder’s priorities for minor use pesticides
• AAFC’s regulatory research products
• PMRA issuing regulatory decisions on minor uses of pesticides submissions (that address the prevention of the growth of the technology gap)

The immediate outcome resulting from the outputs are:

• New minor uses of pesticides are available to growers

The immediate outcome is expected to contribute to the intermediate outcome of:

• Stakeholders implement strategies and use tools to manage changes associated with regulatory modernization and crop protection.

The end outcome of the MUPP is for Agriculture, agri-food and agri-based products sector is able to adapt to a changing regulatory and crop protection environment with scientifically evaluated tools.

Appendix 2 – Performance Timelines and Services Standards for MUPP Submissions

The table below represents performance timelines for minor use pesticides submissions that fall into categories D and C, as referenced in the report in section 3. While performance timelines with specific service standards only apply to the formal applications to add the minor pesticide uses to the label (C.6.3) as they are subject to PMRA's Management of Submission Policy (MOSP), these numbers are also used as internal measures to assess the time it takes to review and approve D.3.1 and D.3.2 submissions.

Appendix 3 – Scope, Evaluation Questions and Data Collection Methods

The evaluation assessed HC's PMRA activities in support of the Minor Use Pesticides Program, which is managed by AAFC, based on two Memoranda of Understanding (MOUs): one signed in 2013, and a renewal in 2018.

The evaluation aimed to answer the following questions:

Evaluators collected and analyzed data from multiple sources. Data collection began in February 2020 and ended in October 2020. Data was analyzed by triangulating information gathered from the different methods listed below. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

Program document and file review – Approximately 140 documents were reviewed to obtain information on all aspects of the MUPP and inform findings related to relevance, effectiveness, and efficiency.

Financial data review – A review of financial data from 2013-14 to 2019-20 was conducted, including a comparison of the planned budget and actual expenditures.

Key informant interviews – Interviews were conducted with 27 internal and external key informants: Program Senior Management and Staff n=10; Federal Partners n=3; Provincial/Territorial Partners n=4; International n=1; and Industry Stakeholders n=9.

As with many evaluations, there were some limitations encountered during the implementation of the selected methods that may have had some implications for the validity and reliability of the evaluation findings and conclusions. The following table provides a summary of the limitations, impacts, and mitigation strategies used to ensure the findings could be used with confidence to guide program planning and decision-making.

Appendix 4 – MUPP Process Submission Flowchart

Figure 3 - Text Description

The Minor Use Pesticides Program (MUPP) considers the expansion of a label for a new minor use of a pesticides based on identified pest management needs. The diagram is a visual representation of the stages and steps involved in the Minor Use Pesticides program's submission and approval process of proposed minor use label expansion applications.

The vertical columns represent the stages pertaining to the submission process with four columns: Sponsor, Pre-submission, Submission, and Registration.

The coloured boxes identify the activities of the parties involved or decision points within each of the columns: the dark and light gray boxes indicate activities that fall outside of the MUPP; the green boxes indicate activities performed by PMRA's MUPP; the yellow boxes indicate decision points; and the red boxes indicate the results of the process.

In the first column, with the title 'sponsor, a pest management need is identified and a letter of support is signed by the registrant. The registrants decide which category of use this application will fall under (A,B, or D). Activities in this column fall outside the work specific to the MUPP. In this column, applicants work with a sponsor to develop a proposal and for AAFC-PMC in collaboration with provincial and forestry minor use coordinators to confirm the need.

Identified in the second column, are the activities that are part of the 'pre-consultation' stage, whereby with support from manufacturers, AAFC and the provinces develop application packages containing any required data for submission to PMRA (classified as a D.3.1 submission).

Once AAFC or the provinces provide all the information required, identified in the third 'submission' column, PMRA then conducts a scientific assessment of the health and environmental risks, as well as a value assessment of the pesticide products included in the submission, resulting in a proposed regulatory decision (classified as a D.3.2 submission). PMRA proposes a regulatory decision if the risks and value are considered acceptable for the proposed use expansion. If the risk and value assessments are deemed to be acceptable, the registrant can then submit a formal application to legally add the minor uses to the pesticide product label (classified as a C.6.3 submission), identified in the fourth 'registration' column.

Appendix 5 – Performance Data of PMRA's Activities in Support of the Minor Use Pesticides Program

Data collected through PMRA's electronic Pesticide Regulatory System database (ePRS) allowed the assessment of the extent to which the MUPP had met the performance targets included in the 2013 and 2018 MOU between AAFC and PMRA, as well as other program performance metrics.

Performance analyses focused on three performance targets: the number of minor use regulatory submissions provided to HC's PMRA, the number of AAFC and provincial minor use submissions reviewed by HC's PMRA, and the number of regulatory decisions that resulted in an increase of sponsor-submitted new pest management tools and minor use registrations available to growers.

Other metrics used by the MUPP to monitor program performance include the C.6.3 submissions. In this step, once PMRA approves an application, registrants can then move forward to activate the process of legally expanding the pesticide label for new crops or pests. As already mentioned, these applications are subject to service standards, based on the number of days it takes to process these applications.