Audit of the Multi-Sectoral Partnerships (MSP) Program to Promote Healthy Living and Prevent Chronic Disease at the Public Health Agency of Canada

Final Report: February 2020

Table of contents

• List of acronyms
• Executive summary
• A: Introduction
  • Program context
  • Program approach
• B: Findings, recommendations and management responses
  • Governance
  • Risk management
  • Internal controls
    • Letter of Intent and proposal review
    • Conflict of interest forms
    • Service standards: LOI
    • Service standards: Proposal
  • Monitoring of program performance
    • Monitoring of recipient performance
    • Contribution agreement reporting requirements
    • Project activity monitoring
    • Recipient audits and site visits
    • Project close-out
  • Sex- and Gender-Based Analysis Plus (SGBA+)
• C: Conclusion
• Appendix A: About the audit
  • Audit objective
  • Audit scope
  • Audit approach
  • Statement of conformance
• Appendix B: Lines of enquiry and criteria
• Appendix C: Scorecard

List of acronyms

BOP

Branch Operation Plan

FAF

Funding Approval Form

HPCDP

Health Promotion and Chronic Disease Prevention

ISHLCD

Integrated Strategy on Healthy Living and Chronic Diseases

LOI

Letter of Intent

MSP

Multi-Sectoral Partnership

PHAC

Public Health Agency of Canada

SGBA+

Sex and Gender-Based Analysis Plus

TBS

Treasury Board of Canada Secretariat

Executive summary

What we examined

The Multi-Sectoral Partnerships (MSP) to Promote Healthy Living and Prevent Chronic Disease Program was launched in 2013, with the aim of advancing innovative solutions to public health challenges. The Program brokers multi-sectoral partnerships and provides co-funding to recipients in order to test and scale up the most promising primary prevention interventions that address common modifiable risk factors for chronic diseases. By engaging with multiple sectors of society, partners can leverage knowledge, expertise, and resources to work towards the shared goal of producing better health outcomes for Canadians.

We examined the management control framework in place to support program delivery from 2014-15 to 2018-19, specifically:

• Governance
• Risk Management
• Internal Controls
• Monitoring of Program Performance
• Sex- and Gender-Based Analysis Plus (SGBA+)

The audit focused on the priority areas of the Integrated Strategy on Healthy Living and Chronic Disease. The MSP Program funded projects under Men’s Health, Canadian Breast Cancer Initiative, Canadian Diabetes Strategy, Cancer, Cardiovascular Disease Program, and Healthy Living Fund programs. It also examined projects, both approved and in development, created under the new Treasury Board of Canada Secretariat pilot on Generic Terms and Conditions, which was officially launched on April 1st, 2017.^{Footnote 1}

Why it’s important

Since 2013, the MSP Program has invested approximately $112 million in support of over 49 projects, which together have leveraged an additional $92 million in investments, involved 400+ partners from across a range of sectors and industries, and used innovative funding models. Innovative and experimental programming can include higher-risk endeavours, yet is expected to lead to sustained approaches to reducing major chronic diseases.

What was found

Overall, we found that the management controls for the MSP Program were generally working as intended; however, there were weaknesses found in controls for the Program’s governance, risk management, and monitoring. We also found that SGBA+ was not sufficiently documented to demonstrate how it was incorporated into relevant processes of the MSP Program. 

Management controls were effective in the following areas:

• The Program appropriately reviewed outputs, outcomes, and budgets contained within the Letters of Intent (LOI) and detailed proposals; and
• Project activities and deliverables, as well as close-out of agreements were, for the most part, appropriately monitored and corrective actions were taken on the results of recipient audits and site visits.

Within the context of Grants and Contributions (G&Cs) management within PHAC, we found management controls were weak in the following areas:

• The Program’s governance did not have fully defined roles and responsibilities for its key activities or relevant stakeholders, lacked a formalized human resource plan, did not have a documented approval process in place at the proposal review stage, was missing terms of reference for the oversight committees, and did not document how monitoring of the Program’s ability to meet strategic objectives was taking place;
• Risk management at the Program level was not being exercised in a consistent manner;
• Service standards were not being met, or did not exist for some Program activities, in order to be able to monitor performance against standards, and the conflict of interest process was inconsistently applied;
• There was no risk-based project monitoring strategy for “innovative funding agreements”; and
• The incorporation of SGBA+ into the MSP Program’s processes was not documented, and Program expectations were not communicated to applicants in relation to target populations.

The areas for improvement noted in this audit report will further strengthen the management practices of the MSP Program and, more broadly, Grants and Contributions management.

A: Introduction

Program context

1. According to the World Health Organization (WHO), non-communicable diseases (commonly known as “chronic diseases” in Canada) are increasingly complex health challenges. In Canada, more than a fifth of the population over the age of twenty is living with at least one major chronic disease, such as cancer, diabetes, cardiovascular disease, or chronic respiratory disease. Physical inactivity, unhealthy eating, tobacco use, the problematic use of alcohol, and exposure to unhealthy built environments are all common modifiable risk factors for chronic disease.^{Footnote 2} Obesity is also considered to be a key driver of chronic diseases in Canada, where one in three children are overweight or obese, and over one in four adults are obese.
2. Over 150,000 Canadians die annually from diseases that are preventable. Chronic diseases are also responsible for a high rate of morbidity, associated reductions in quality of life, and negative impacts on communities and the economy. Chronic diseases and other illnesses cost the Canadian economy $190 billion annually, including $122 billion in indirect income and productivity losses, and $68 billion in direct health care costs.^{Footnote 3}
3. In May 2012, the World Health Assembly endorsed the first global target to reduce premature mortality from non-communicable disease by 25 percent by 2025. Member states also came to an agreement on a comprehensive global monitoring framework.^{Footnote 4} The 2030 Agenda for Sustainable Development increased this global target to one-third (33 per cent) by 2030.
4. In 2005, the Public Health Agency of Canada (PHAC) received $300 million for five years, and $66.7 million every year in ongoing funding, for the Integrated Strategy on Healthy Living and Chronic Disease (ISHLCD), which represents PHAC’s foundation in the area of healthy living and chronic disease prevention. The aim of the ISHLCD framework was to ensure that Canada had an integrated approach to addressing major chronic diseases by focusing on common modifiable risk factors for chronic diseases, as well as through complementary disease-specific work. The framework consisted of three pillars: promoting health, preventing chronic diseases by minimizing risk, and early detection and management of chronic diseases. The Centre for Chronic Disease Prevention and Health Equity (CCDPHE), within the Health Promotion and Chronic Disease Prevention (HPCDP) Branch, leads PHAC’s chronic disease prevention activities.^{Footnote 5}
5. Grants and Contributions are jointly managed between the Centre for Grants and Contributions (GGC) and program staff. The Transfer Payment Business Process developed by PHAC outlines the key steps in G&Cs operations and management. The process is more fully described in the G&Cs Business Management Model (BMM), which also documents the accountability and responsibilities of programs and the CGC for key aspects of the G&Cs life cycle.
6. The CGC, located within Office of the Chief Financial Officer, supports the administration of PHAC’s G&Cs programs. CGC is responsible for the administration of contribution agreements, payments, financial oversight, and recipient audits. CGC developed a set of Standard Operating Procedures (SOPs), information, and tools to assist staff across the Agency in making informed decisions concerning the management and monitoring of G&Cs programs and agreements.
7. Program staff at headquarters is responsible for overall program management, which includes stakeholder engagement, program design, assessment of funding requests, funding decisions, monitoring of the agreement and deliverables, and assessment of program outcomes.
8. Funding for the Healthy Living and Chronic Disease Prevention – Multi-Sectoral Partnerships (MSP) Program is provided through the ISHLCD. PHAC’s MSP Program was launched in 2013, with the aim of advancing innovative solutions to public health challenges. The Program provides co-investments to partners for testing and scaling up the most promising primary prevention interventions, with the goal of improving health overall, while reducing health inequalities among population groups where necessary.

Program approach

9. Overall, the Program focuses on a multi-sectoral approach to address complex social issues, whereby each project must include partnerships from both the private and not-for profit sectors, such as governmental organizations, academia, industry, and communities. By engaging multiple sectors of society, partners can leverage knowledge, expertise, and resources in working towards the common goal of producing better health outcomes for Canadians. Since 2013, the MSP Program has invested approximately $112 million in support of over 49 projects, which leveraged an additional $92 million in investments and involved over 400 partners from across a range of sectors and industries.
10. The Program uses a continuous intake approach for project submissions. Rather than using a specific proposal deadline, the Program uses an open solicitation process and a two-step review process for incoming detailed proposals. The first step is the receipt and review of Letters of Intent (LOI), followed by the receipt and review of detailed proposals. This intake approach generally involves a high degree of co-creation between program officials and project proponents.
11. Consistent with the multi-sectoral approach, the Program also requires project partners to secure matched funding, identified as a ratio of 1:1, of financial or in-kind contributions from non-taxpayer funded sources to be eligible for funding from the Program. There is no other program within the Government of Canada that has both a continuous intake with a two-step review process, and a multi-sectoral approach specifically focusing on private sector matched funding.
12. The MSP Program also makes use of outcome-based funding models in its project design, as set out under the Treasury Board of Canada Secretariat’s (TBS) new Generic Terms and Conditions for Innovative Uses of Transfer Payments Pilot.^{Footnote 6} Such funding models focus on innovative financing tools, based on the achievement of concrete results and goals, rather than traditional funding that is based on tasks and activities. The Program also identified a series of physiological and psychological measures related to common modifiable risk factors for chronic diseases (e.g., blood pressure, body mass index), which are used as performance indicators to tie payments to recipients’ successful achievement of project results.

B: Findings, recommendations and management responses^{Footnote 7}

Governance

13. Governance is a set of rules and practices by which management ensures coordination, leadership, and accountability. Ultimately, the application of good governance contributes to the effective and efficient realization of strategic and organizational goals.
14. We expected to find appropriate governance in place and operating effectively to provide strategic and operational direction for management of the MSP Program.
15. We found that the Centre for Chronic Disease Prevention and Health Equity’s (CCDPHE) Strategic Plan and the Health Promotion and Chronic Disease Prevention’s (HPCDP) Branch Operational Plan (BOP) clearly articulated the Program’s strategic direction. We found the Program had detailed annual operational and resource plans that were integrated into the BOP.
16. The Program identified challenges related to staffing capacity in terms of both personnel numbers and skill sets, and provided examples of how these challenges were currently being addressed. However, we found the Program did not have a human resources (HR) plan. The Program’s annual operational plan, which was integrated into the 2018-19 BOP, included some components of an HR plan, such as training costs and plans for staffing in that year. However, it did not include succession planning, knowledge transfer, and workforce gap analysis. The Branch committed to, in the most recent BOP, developing a branch-level HR plan and a capacity and gap assessment. While these initiatives have yet to be completed by the Branch, the Program should align its HR strategy with that of the Branch to ensure effective use of resources and ensure the Program’s ability to attract and retain the talent needed to achieve its operational objectives and meet its unique and innovative mandate.
17. The Program, as part of the Partnership and Strategies Division, went through a large reorganization in 2018, resulting in the Program operating through four new teams that were established in September 2018. Each team focused on the following different program areas: policy, partnerships, project operations, and impact analysis. Given this large-scale change, the teams’ roles and responsibilities were not always clear. For example, the performance measurement unit, which was in a different division than the Program, had an important role in the review of detailed proposals, related to establishing project and program performance targets, yet their role was not identified in how they complement the four teams, or their role within the Letters of Intent (LOI) and proposal review process. We found that the roles and responsibilities of the project operations team, by contrast, which dealt mostly with contribution agreement (CA) monitoring, were the most understood and followed. By not having defined roles and responsibilities, there is a risk of confusion and duplication of roles between the four teams, leading to program inefficiency.
18. The Program had a Review Committee that made the final decision to accept or reject LOIs. We reviewed minutes from committee meetings that documented the discussions and decisions taken to approve or reject the LOIs. Once this review committee approved an LOI, the applicant was invited to submit a detailed proposal based on the initial LOI. There were no terms of reference governing the Review Committee’s responsibilities.
19. We expected to find a similar process in place for the review of the detailed proposals. While we could track the back-and-forth interactions between the Program and applicant relating to proposal information, a documented approval process or concurrence was not present prior to moving the proposal to the Funding Approval Form (FAF) stage. The FAF stage is when key program and branch officials, along with the Agency’s Chief Financial Officer, sign off on the decision to provide funding to an applicant. Having an oversight mechanism, such as a Review Committee at this stage in the process, would provide a forum for discussions to be held and decisions to be made and documented on a specific project before moving to the FAF stage. In addition, having an independent reviewer, or other mechanism like this committee, could help deter any actual or perceived conflicts of interest.
20. Finally, we could not determine how the Program ensured oversight of its strategic outcomes, specifically that it was meeting its strategic objectives of experimentation and scaling up of projects, developing strategic partnerships, and ensuring the funding of innovative projects. Oversight of strategic partnerships was intended to be covered through the Partnership Check-up process, but this process was still in development during the examination phase of this audit. Without documented evidence of such oversight, it is difficult to determine how the Program monitors and makes the changes necessary to help them achieve their strategic goals.

Recommendation 1

The Vice-President, Health Promotion and Chronic Disease Prevention Branch, should:

• develop a Program-level human resource plan
• finalize roles and responsibilities for the four teams, as well as that of the Performance Measurement team
• develop terms of reference for existing oversight committees
• document the decisions stemming from the Program’s proposal review stage
• monitor the progress of the Program’s achievement of strategic objectives

Management response

Risk management

21. Risk management is an integral component of good management that supports organizations in making informed decisions for allocating resources, mitigating threats, and proactively capitalizing on opportunities. Risk management is not a one-size-fits-all approach, but rather a process that is tailored to the organizational environment, including its mandate, structure, operations, and related constraints.
22. We expected to find a risk management process in place where management identifies, assesses, and manages the risks that may preclude the achievement of its objectives.
23. As highlighted in the November 2018 - Audit of the Management of Grants and Contributions at Public Health Agency of Canada, the Agency focused its risk management processes and activities at the individual or agreement levels, and not at the program level. The following recommendation was made in the audit:
    
    “The Chief Financial Officer, supported by the Office of Strategic Policy and Planning, collaborates with the Vice-Presidents of the HPCDP Branch, HSIB, and IDPC Branch to develop a formalized process for documenting, monitoring, updating, and reporting risks and risk management strategies at the G&Cs program level, in accordance with the IRM, and to be implemented within the Vice-President’s respective program area.”

24. In line with the finding which led to the above recommendation related to the Agency’s G&Cs program functions as a whole, we found that the MSP Program, similarly, did not have a risk management process in place where management identifies, assesses, and manages possible program-level risks that may preclude the achievement of its objectives. The Program identified certain risks in an ad hoc manner, and was in the process of addressing them. For example, the Program recently developed a risk framework that would allow it to conduct its own risk assessments and establish mitigation measures to ensure identified risks were managed. The Program identified some risks, but did not assess their level, nor did it assign related mitigation strategies. Without an established risk management process, the Program may not be able to identify all major risks, nor implement a strategy to address them, which may affect it meeting its objectives.
25. Overall, the Program conducted informal risk management activities, and efforts were being made to more systematically identify, assess, and manage program risks. This process should be finalized and incorporated into the Agency-wide risk management process.

Recommendation 2

The Vice-President, Health Promotion and Chronic Disease Prevention Branch, should implement the Program’s specific risk management framework, while ensuring that it is aligned with the work being done to address Recommendation 3 of the Audit of the Management of Grants and Contributions.

Management response

Internal controls

Letter of Intent and proposal review

26. The Program’s review process consisted of two principal stages. The first was an initial screening to ensure the LOI was complete. Once this was done, a detailed review of the LOI took place. After the LOI had been reviewed and accepted, the applicant was then sent an Invitation to Submit a Funding Request (ISFR), which was an in-depth version of the LOI. This led to the second review stage, where the Program reviewed the proposal in order to determine if the project should be funded. During this stage, the Program might have co-created projects with applicants, by helping with the design and with identifying potential investors, in order to ensure that the project reaches an optimal state and secures all necessary matched funds before program funding is approved.
27. Appropriate LOI and proposal review, in accordance with the established process, ensures that applicants receive a fair and accurate assessment of their project idea. This includes appropriate review of the project’s target population and specific activities to ensure they are in alignment with the Program’s objectives. As part of these assessments, each reviewer is required to sign a Conflict of Interest Form (COI) before reviewing either an LOI or a detailed proposal to ensure an unbiased and accurate assessment of the project. This process requires reviewers to self-identify any real, apparent, or potential conflict of interest, allowing management of the conflict in a manner that upholds the public interest.
28. Within the context of the Program’s matched funding requirement and open solicitation process, the efficient review of LOIs and detailed proposals is an important factor for ensuring that external partners remain committed to providing applicants with matched funds or other supports. It also enables applicants and partners to plan projects better, to ensure that detailed project proposals remain relevant and can be advanced in a timely manner, and helps the Program maintain its image as a reputable partner. Communicating and adhering to established service standards for applicants can be beneficial in ensuring the efficient review of LOIs and detailed proposals. Service standards are an important element to ensuring the quality of the services provided, as they help clarify expectations for clients and employees, enable performance management, and support client satisfaction.
29. We expected to find a process in place for the efficient and appropriate review and approval of LOIs and detailed project proposals.
30. In order to determine if the review of LOIs and detailed proposals was carried out in an appropriate and efficient manner, we tested a sample of LOIs and detailed proposals that had been rejected or accepted, as well as detailed proposals under review. Details on the population and samples are outlined in Table 1 below.

Table 1: Number of LOIs and Detailed Proposals Reviewed

Type	Population (2014 to 15) to December 31, 2018	Sample
Footnote * LOIs were not found on file for some projects, as the Program had directly asked the applicant to submit a detailed proposal instead of an LOI.  Return to footnote * referrer
Total samples tested: 74 LOIs reviewed
Rejected LOIs	130	50
Accepted LOIs	38Footnote *	24
Total samples tested: 43 proposals reviewed
Accepted proposals with a Contribution agreement	42	27
Proposals under review when testing started	7	5
Proposals not accepted (no  contribution agreement)	16	7
Proposals under the new TB Generic Terms and Conditions	4	4

31. Through a review of these samples, we found that the controls for reviewing the LOIs and detailed proposals were generally followed, with minor discrepancies that were presented to management for correction. 

Conflict of interest forms

32. We reviewed the COI process for both the review of LOIs and of detailed proposals that the Program had implemented, in addition to the G&Cs Standard Operating Procedures (SOPs) COI process. We noted that PHAC employees involved in the review of these documents did not always follow the Program’s own policy guidelines of signing a COI disclosure form each time they conducted a review. The Program stated that they required the use of an external reviewer to take part in the review process, as a further measure to support impartiality during LOI and Proposal review. We noted that this requirement was not always followed.
33. Given the design of this innovative program, project proponents must collaborate and obtain cash or in-kind support from various public and private sector entities across Canada. Program staff are also involved in the co-creation of projects. This involvement in the project through co-creation, while innovative, creates the risk of a real or perceived conflict of interest. Requiring employees conducting reviews to sign a declaration of COI at the time of the review would help reduce this risk.

Recommendation 3

The Chief Financial Officer should update the Grants and Contributions Standard Operating Procedures related to conflict of interest requirements for dealing with higher risk projects such as innovative funding agreements or innovative projects.

Management response

Service standards: LOI

34. Service standards are timelines that are established and committed to, usually based on predetermined norms for completing a certain step or process. They allow for improved performance measurement capabilities and planning. The Program established a service standard of 45 calendar days for the LOI review process and posted this on their public website. No service standards exist for the proposal review phase.
35. As noted in Table 3, we found that for the past four-and-a-half years, the average review time of 68 calendar days was above the set standard of 45 days. We noted that during the same period, the number of LOIs received and reviewed decreased from 58 in 2014-15 to 26 in 2017-18, yet the average review time for the LOI did not change significantly. Based on this information, we can conclude that the Program should review their processes to establish realistic service standards that are communicated through the Program’s website.

Table 2: Average number of days to review a letter of intent

Fiscal year	Average days to review a LOI by fiscal year (total average: 68 days)	Number of LOIs sent to the Program each fiscal year
Footnote * Average review calendar days after initial screening  Return to footnote * referrer Footnote ** This average of 84 calendar days includes one LOI review that took the most number of days to review, at 204 days, and an LOI that took the least number of days to review, at 33 days. At least four LOIs included in this average were only approved conditionally, based on budget considerations.   Return to footnote ** referrer
2014 to 15	69Footnote *	58
2015 to 16	85Footnote *	46
2016 to 17	52Footnote *	27
2017 to 18	65Footnote *	26
April 2018 to December 2018	84Footnote **	24

36. Service standards are an important element for measuring the quality of services provided and ensuring client satisfaction. Given that the Program has, in most cases, not met its advertised 45 calendar day service standard, there is the risk that the Program’s delays in reviewing LOIs and detailed proposals may negatively affect an applicant’s ability to efficiently plan their projects and successfully negotiate for funds from external partners. It could also prolong the proposal review process even after an LOI has been approved.

Service standards: Proposal

37. We found that the Program did not monitor its performance, nor did it have service standards for the proposal review stage. Through our examination of detailed proposals that were reviewed over the period of 2016-17 to 2018-19, we noted that the average review time from receipt to issuance of the acceptance letter was 374 calendar days, with times ranging from 80 days up to 1,227 days. The factors that may have contributed to the average review time of 374 calendar days included: the finalization of the budget and securing of matched funds which took the longest time, averaging 316 days; the Program’s finalization of the FAF, averaging 78 days; and the completion of review forms, averaging 48 days.  

Recommendation 4

The Vice-President, Health Promotion and Chronic Disease Prevention Branch, should validate the Branch’s service standards, and align the Program’s business processes and monitor the Program’s achievement against these standards.

Management response

Monitoring of program performance

Monitoring of recipient performance

38. The Directive on Transfer Payments requires that departmental management exercise a level of monitoring that is commensurate with the risks of the programs, the value of the funding in relation to administrative costs, and the risk profile of the recipients. Ongoing monitoring of recipient performance enables re-assessment of risk based on changes in the recipient environment and remediation of identified deficiencies, thus contributing to the overall success of the project.
39. We expected that recipient reporting and close-out of agreements were appropriately monitored, and that the results of recipient audits and site visits were used to take appropriate corrective action when required. 

Contribution agreement reporting requirements

40. The frequency and nature of recipient reporting requirements are outlined in Appendix C of Contribution Agreements (CA). Reporting requirements include a variety of financial documentation, such as audited financial statements and general ledgers, and non-financial documentation, such as quarterly progress reports and annual progress reports.
41. We expected to find reporting requirements, as outlined in Appendix C, that would allow program managers to appropriately monitor recipient progress.
42. We reviewed whether recipient reporting requirements were included in the Grants and Contributions Information Management System (GCIMS), which is managed by the Centre for Grants and Contributions, to allow for appropriate and timely monitoring. We noted that six declaration of income forms and three progress reports were not on file for the 27 agreements reviewed. Furthermore, an instance was identified where a payment was issued to the recipient prior to their signing the CA, even though the Program’s policy states that costs cannot be incurred before a CA is signed.
43. While several reporting requirements were not collected, the progress of project activities was continuously monitored using other methods (refer to the next section, Project Activity Monitoring). This provided assurance that payments were made based on expected project progress. Overall, increased effort is required to ensure that all reports are submitted for review on a timely basis to support management in fulfilling their responsibilities.

Project activity monitoring

44. The Treasury Board of Canada’s Directive on Transfer Payments requires that departmental managers ensure that the level of monitoring of recipients reflects an assessment of program-specific risks, value of funding in relation to administrative costs, and the risk profile of the recipients.
45. We expected to find that project activities were appropriately monitored, based on the level of risk identified in their monitoring strategy which is based on Agreement/Recipient Risk Assessment Tool (ARRAT) results.
46. We found that a risk-based monitoring strategy for grants and contributions was in place, with applicable tools and that was applied to all projects. However, no specific strategy or tools were in place for ‘innovative funding agreement’ projects within the MSP Program, even though the latter were more complex, with a higher degree of risk. Through a review of files, we found there was no standardized process to monitor recipients. There were varying monitoring tools on file for each case, but no standard set of documents. The risk assessments on file identified which recipients should undergo a recipient audit and site visit, with no indication of the frequency of the monitoring. As a result, it was difficult to determine if the level of monitoring was appropriate and aligned with the risk profile of the recipient. Of the 27 files reviewed, there were three that had no monitoring or documented communication with recipients.

Recipient audits and site visits

47. We expected to find that the results of recipient audits and site visits were used to identify issues and take corrective action.
48. Recipient audits were conducted for all high-risk recipients. We reviewed seven recipient audits and found that the results of these audits were used to identify recoverable amounts owed to PHAC and, in all cases, those amounts were recovered. We noted that the Program introduced clauses in recently-signed CAs that allow for recipient audits to be used to verify that the matched cash funding from partners, as identified in the partner support letters, was both received and spent on the PHAC-funded project.
49. We found that site visit plans were developed for each fiscal year, based on three strategies. However, the Program stated that plans were not always followed due to financial constraints and staff shortages. Of the 25 site visits planned for the period in scope (2016-17 to 2018-19), we found that seven had been conducted as planned, and five had been conducted on an ad hoc basis. Of these 12 site visits, four did not have Project Monitoring Checklists (PMC) on file as required. Special issues or recommendations were not identified in the PMCs that were completed, and therefore no corrective actions were required following the site visits. 

Project close-out 

50. We expected to find that all final reports were obtained from the recipient before a project was closed out, as required in the CAs. We also expected that any accounts receivable were set up by the Program and paid by the recipient before a project was closed out.
51. Of the 27 projects tested, we found that all of the agreements had been closed out, with some minor control weaknesses. We noted that two recoverable amounts (accounts receivables) were identified, set up, and paid by the recipient before their project was closed, in line with expectations.
52. The minor control weakness noted above is related to deliverables that are required as part of the CA reporting plan and need to be obtained before a project is closed.  We found that three projects out of the 27 sampled did not have final reports on file, two projects did not have their evaluation reports, and one project did not have its final accounting of expenditures.
53. Overall, we found that the reporting and close-out of agreements were appropriately monitored, despite some control weaknesses. Although the Program monitored project activities throughout their life cycle, there was no risk-based monitoring strategy for the monitoring of complex innovative funding agreements, nor was there a program risk tolerance level that could help with the establishment of a monitoring strategy. We also found that, while agreements were monitored and closed out appropriately, there was room for improvement in ensuring that all recipient documentation required under the agreements was obtained. Corrective action was taken, as identified in the recipient audits and site visits, but there was room for minor improvement in the monitoring process for site visits.

Recommendation 5

Recommendation 5: The Chief Financial Officer should update the Grants and Contributions Standard Operating Procedures to include a risk-based project monitoring strategy for innovative funding agreements that is structured around results and outcomes, rather than activities and outputs, with payments tied to the achievement of pre-determined performance expectations.

Management response

Recommendation 6

Based upon the newly-developed, risk-based project monitoring strategy for innovative funding agreements, the Vice-President, Health Promotion and Chronic Disease Prevention Branch, should implement a risk-based project monitoring strategy for innovative funding agreements that aligns with the MSP Program’s overall risk tolerance.

Management response

Sex- and Gender-Based Analysis Plus (SGBA+)

54. SGBA+ is an analytical process used to assess how diverse groups of women, men, and non-binary people may experience policies, programs, and initiatives. Many other identity factors may also play a role, such as race, ethnicity, religion, age, and mental or physical disability.
55. We expected to find that SGBA+ had been incorporated into the life cycle of the Program’s activities, as outlined in the Agency’s SGBA+ guidance.
56. We found that there has been greater effort to address specific and identified sex and gender gaps, as applicants were asked to identify their project’s target populations in their detailed proposals, and the Program’s staff members have all completed mandatory SGBA+ training. We did not find evidence that SGBA+ had been integrated in the Program’s processes; for example, that SGBA+ factors were considered during the proposal review stage, or that SGBA+-related criteria were included in the reporting structure for project outputs and outcomes.
57. While SGBA+ has been completed for healthy eating (December 2017), for smoking (October 2018), and for sedentary behaviour and physical inactivity (November 2018), the results of the analysis have yet to be converted into a program-level analysis that can be used to assess SGBA+ in future project proposal submissions.
58. We did not find evidence of inclusion of these SGBA+ principles in the analysis completed during the review of target populations, as identified in project Letters of Intent (LOI) or detailed proposals. Nor did the Program have set criteria or expectations for assessing the applicant with regards to SGBA+. Finally, we did not find evidence that the Program had made applicants aware of SGBA+ reporting requirements once their proposal had been approved. As such, SGBA+ indicators were not integrated into the reporting structure for project outputs and outcomes, including progress reports and annual reports, thus making it difficult for the performance measurement team to consistently gather and report on this information.
59. Although most interviewed project applicants, recipients, and partners were aware of the Program’s website, they had difficulty finding relevant information on priorities, strategic directions, and target populations related to SGBA+, and would have liked a clearer indication of the types of projects the MSP Program seeks to fund.
60. While the Program is committed to encouraging experimentation and avoiding limitations for the intake of innovative projects, we did not find sufficient documented evidence of how SGBA+ was incorporated into relevant MSP Program processes, or how project funding decisions addressed common modifiable risk factors for chronic disease in the most vulnerable populations.

Recommendation 7

The Vice-President, Health Promotion and Chronic Disease Prevention Branch should integrate SGBA+ into the Program’s processes, and ensure that related expectations are communicated to applicants.

Management response

C: Conclusion

Overall, we found that the management controls for the MSP Program were generally working as intended; however, there were weaknesses found in controls for the Program’s governance, risk management, and monitoring. We also found that SGBA+ was not sufficiently documented to demonstrate how it was incorporated into relevant processes of the MSP Program. 

Management controls were effective in the following areas:

• The Program appropriately reviewed outputs, outcomes, and budgets contained within the Letters of Intent (LOI) and detailed proposals; and
• Project activities and deliverables, as well as close-out of agreements were for the most part appropriately monitored and corrective actions were taken on the results of recipient audits and site visits.

Within the context of Grants and Contributions (G&Cs) management within PHAC, we found management controls were weak in the following areas:

• The Program’s governance did not have fully defined roles and responsibilities for its key activities or relevant stakeholders, lacked a human resource plan, did not have a documented approval process in place at the proposal review stage, was missing terms of reference for the oversight committee and did not monitor whether the Program was achieving its strategic objectives;
• Risk management at the Program level was not being exercised in a consistent manner;
• Service standards were not being met, or did not exist for some Program activities, in order to be able to monitor performance against standards and the conflict of interest process was inconsistently applied;
• There was no risk-based project monitoring strategy for “innovative funding agreements” ; and
• The incorporation of SGBA+ into the MSP Program’s processes was not well documented, and Program expectations were not communicated to applicants in relation to target populations.

The areas for improvement noted in this audit report will further strengthen the management practices of the MSP Program and, more broadly, Grants and Contributions management.

Appendix A: About the audit

Audit objective

The objectives of this audit were to provide reasonable assurance that management controls were working as intended, and that Sex- and Gender-Based Analysis Plus was being integrated into the life cycle of the Program.

Audit scope

The scope of this audit focused on the structures, processes, and activities in place from April 2014 to December 2018, and included the examination and assessment of governance mechanisms, risk management, internal controls over the application phase, as well as monitoring of the Program’s performance.

Audit approach

The audit was conducted at the Public Health Agency of Canada’s Headquarters (HQ) and included processes in place and activities carried out by the MSP program and the Office of the Chief Financial Officer (OCFO).

The principal audit procedures included, but were not limited to:

• walkthroughs and interviews with PSD and CGC (OCFO);
• review of documentation, policies, standards, guidelines, and frameworks related to grants and contributions;
• detailed testing of related controls over MSP activities; and,
• analysis of findings from interviews, enquiries, document reviews, and detailed testing.

Statement of conformance

This audit was conducted in conformance with the International Standards for the Professional Practice of Internal Auditing, as supported by the results of the Office of Audit and Evaluation's Quality Assurance and Improvement Program.

Appendix B: Lines of enquiry and criteria

Audit of the Multi-Sectoral Partnerships Program to Promote Healthy Living and Prevent Chronic Disease

Criteria	Sub-criteria
Appropriate governance is in place for the Program.	1.1 There is an operational and human resource plan for the Program that aligns with the centre and branch’s strategic objectives. 1.2 Accountabilities and responsibilities are established and followed for the Program’s management and delivery.  1.3 The Program is supported by appropriate oversight committees, with documented terms of reference.
Management identifies, assesses, and manages the risks that may preclude the achievement of its objectives.	2.1 Internal and external risks related to program activities are identified and assessed. 2.2 Risk mitigation strategies are identified and implemented.
Efficient and appropriate review and approval of project proposals takes place.	3.1 Proposals are reviewed against appropriate criteria in an efficient manner. 3.2 The sustainability of outputs and outcomes, and compliance to program terms and conditions are challenged. 3.3 The project budgets are challenged. 3.4 Funding is approved against appropriate criteria in an efficient manner.
Recipient reporting and close-out of agreements are appropriately monitored.	4.1 Funding agreements are monitored and closed out against appropriate criteria. 4.2 The results of the recipient audits and site visits are used to take corrective action.
Sex- and Gender-Based Analysis Plus is incorporated into the life cycle of the Program.	5.1 The Program uses SGBA+ to determine the target populations. 5.2 The target populations are communicated to applicants. 5.3 Proposals are approved if they clearly identify how they meet the needs of the Program’s SGBA+. 5.4 For approved recipients, the inclusion of sex and gender is considered by the Program as they monitor project performance and outcomes.

Appendix C: Scorecard

Audit of the Multi-Sectoral Partnerships (MSP) Program to Promote Healthy Living and Prevent Chronic Disease at the Public Health Agency of Canada

Criterion	Risk ratingFootnote 8	Risk remaining to Program objectives without implementing recommendation	Recommendation number
Definitions of risk levels: Minimal risk Minor risk Moderate risk Significant risk Major risk
Appropriate governance is in place for the Program.	3	The attraction and retention of qualified staff for current and future needs may be at risk without appropriate Program HR planning; Confusion of roles and responsibilities after a reorganization risks program efficiency; There is a risk of oversight committee members not understanding their roles and responsibilities due to the absence of or incomplete terms of reference; Senior management may not understand how and what decisions were made in the review of project proposals due to undocumented decisions; and The Program may not be able to meet its objectives as it is not monitoring the achievement of its strategic objectives.	1
Management identifies, assesses, and manages the risks that may preclude the achievement of its objectives.	2	There is a risk to the Program’s costs, performance, and reputation, as well as possible delay or inefficiency due to the absence of a program-level risk management process.	2
Efficient and appropriate review and approval of project proposals takes place.	3	There is an increased risk of reputational loss or impact to the Agency’s reputation due to the real or perceived conflict of interest occurring within the Program, due to the inconsistent application of a robust conflict of interest process.	3
Efficient and appropriate review and approval of project proposals takes place.	3	There is a risk to the Program’s reputation from the possible negative effect on an applicant’s ability to efficiently plan their projects and successfully negotiate for funds from external partners due to the Program’s inability to respect service standards, or to not have established service standards for some processes that leads to delays in the Program reviewing LOIs and detailed proposals.	4
Recipient reporting and close-out of agreements are appropriately monitored.	3	There is a risk of potential misuse or ineffective use of funds or not meeting the grant and contribution program objectives, due to inconsistent or inadequate monitoring of innovative funding recipients, as there is not a standard operating procedures in place for risk-based project monitoring for innovative funding agreements that, by their nature, are high-risk agreements.	5
Recipient reporting and close-out of agreements are appropriately monitored.	3	There is a risk that there is inconsistent or inadequate monitoring of all recipients, leading to potential misuse or ineffective use of funds, or not meeting the Program’s objectives, due to there not being a risk-based monitoring strategy for innovative funding agreements.	6
Sex- and Gender-Based Analysis Plus is incorporated into the life cycle of the Program.	4	There is a risk of not addressing common modifiable risk factors for chronic disease in the most vulnerable populations, due to the Program not fully incorporating SGBA+ into relevant processes. The Program risks inefficiency in its review process and not targeting identified populations by not communicating to applicants the priority of the common modifiable risk factors for chronic disease in the most vulnerable populations.	7

Related links

• Management Response and Action Plan: Audit of the Multi-Sectoral Partnerships (MSP) Program to Promote Healthy Living and Prevent Chronic Disease
• Multi-sectoral Partnerships to Promote Healthy Living and Prevent Chronic Disease