ARCHIVED - Integrated Strategy on Healthy Living and Chronic Diseases - Community Based Programming Functional Component

Key Findings

This section presents key findings from the Diabetes CBP formative evaluation in relation to three evaluation issues: relevance; design and delivery, and success/progress.

3.1 Relevance

Does the Component continue to be consistent with departmental-wide priorities, and does it realistically address an actual need?

In summary, this evaluation found there is general support for PHAC’s role in the prevention of chronic disease, including diabetes.  However, the findings indicate a need to define PHAC’s role vis-à-vis that of P/T governments in addressing the management of diabetes, including support for community-based programs in this area.  There was also evidence of potential overlap and/or duplication across ISHLCD Functional Components, as well as within the CBP Functional Component.

The latest data from the National Diabetes Surveillance System (NDSS) indicate a rapidly increasing prevalence of diabetes in Canada, and illustrate the pan-Canadian nature and magnitude of this challenge.  In 2005-06, approximately 1.9 million Canadians of all ages, or about 5.9% of the population, had been diagnosed with diabetes, up from 1.3 million or 4.2% in 2000-01.  It is also estimated that a substantial proportion of Canadians live with undiagnosed diabetes.  People living with diabetes and their families suffer a reduced quality of life and an increased likelihood of complications, as well as bear significant financial costs associated with medication and other supplies.  The increase in the incidence of diabetes is largely associated with an aging, inactive and overweight population.  By 2011, the number of Canadians with diagnosed diabetes is expected to be about 2.6 million, presenting a serious challenge to society as a whole (Public Health Agency of Canada, 2008b; Public Health Agency of Canada, 2008a).

Continued Need for the Community-Based Programming and Community Capacity Building Functional Component

Diabetes CBP is designed to address the following priority areas: 1) providing community-based health promotion and diabetes prevention programs; 2) providing accessible health services for the prevention of diabetes; and 3) conducting research, evaluation and supporting knowledge exchange.Footnote 5

The Program is based on the population health approach, which recognizes the role of broad determinants of health, as well as the importance of addressing the needs of all members of the population in a variety of ways.  PHAC has identified population health as a key concept and approach for policy and program development aimed at improving the health of Canadians.  Based on this approach, Diabetes CBP encourages collaboration among stakeholders in all sectors and at all levels in addressing the prevention and management of diabetes.  Diabetes CBP is also designed to support community-based programs that are tailored to the unique needs of the community.

The information collected through this evaluation indicates there is general support for the use of population health approaches in diabetes prevention and management.  The need to develop strategies and activities for diabetes prevention grounded in this approach was confirmed through the document review. Also, there was general recognition among stakeholders consulted as part of the recent Policy Review that addressing social and environmental determinants of health is an important aspect of managing diabetes and preventing chronic diseases in general (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b). Stakeholders emphasized the importance of supporting community-based programs to promote healthy living and diabetes prevention at the grassroots level.

Role of PHAC in Chronic Disease Prevention and Control

There was clear support for PHAC’s role in diabetes prevention.  Little evidence was found to support the transfer of elements of CBP to other sectors or levels of government.  Several high-profile and foundational national studies have noted the important leadership role the federal government can and must play in the prevention of chronic disease, including diabetes (Commission on the Future of Health Care in Canada, 2002; National Advisory Committee on SARS and Public Health chaired by Dr. David Naylor, 2003; Standing Senate Committee on Social Affairs et al., 2002).Footnote 6  Most stakeholders consulted through the Diabetes Policy Review identified the prevention of diabetes as an area where PHAC should focus its efforts, especially given this area has not been a focus at the P/T level (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

However, there was some disagreement among the representatives of P/T health departments/ ministries consulted through the Diabetes Policy Review regarding the appropriate role of PHAC in the management of diabetes, including supporting community-based programs.  Some provinces, such as Ontario and New Brunswick, were of the view that PHAC should play a greater role in supporting community-based programs, particularly those related to self-management.  Others, such as Alberta and Manitoba, maintained that PHAC has a role in only limited areas, namely social marketing, research funding, and knowledge development and exchange.  The latter further indicated that federal funding in communities without consulting P/T governments is counterproductive and presents challenges to the work being done at the P/T level (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

Overlap/Duplication across Functional Components and within the CBP Functional Component

This evaluation found evidence of potential overlap/duplication between KDED and CBP at the national level, which is largely a result of lack of clarity regarding each Component’s expected roles under the ISHLCD.  A program representative indicated that while community-based programming is an integral aspect of KDED, the mechanism by which these two Functional Components connect is not clearly defined. Another program representative also noted a potential overlap between the two Components at the national level.  Nationally, KD&E in diabetes appears to have focused on knowledge development to support evidence-based pre-diabetes screening interventions as well as risk factor assessment, with less focus on knowledge transfer.  At the same time, national projects under Diabetes CBP have also supported knowledge development, including the identification of best and/or promising practices.  In the regions, KDED has largely played its expected role under the ISHLCD; that is, providing knowledge that informs CBP.  For example, in the Alberta Region, work done by KDED, such as an analysis of the gaps in knowledge, has greatly informed activities under CBP.  The Second Implementation Review found that most PHAC staff felt that in a matrix structure, some overlap and duplication among Functional Components would be desirable to ensure the proper intersection of crosscutting functions. Staff members did express concern with the lack of coordination and potential duplication where Functional Components intersect (Public Health Agency of Canada, 2008c).

There was also evidence of potential overlap/duplication within the CBP Functional Component, between Diabetes CBP and the Healthy Living Fund.  The expected results of the Healthy Living Fund include increased capacity within the healthy living stakeholder community at all levels in areas of: knowledge development; partnerships and networking; creating supportive environments and conditions for healthy living; and policies, programs and services.  Approximately half (21 out of 41) of the projects funded through Diabetes CBP in 2006-07 were aimed at addressing the prevention of diabetes among high-risk groups, and at least 71% (or n=15) of these projects were designed to address the promotion of healthy living (healthy eating and physical activity). Program representatives interviewed noted the two programs had been working very closely to leverage the program resources and help achieve their expected outcomes.  However, stakeholders consulted as part of the Diabetes Policy Review unanimously pointed to the lack of clarity for different funding programs under the ISHLCD as a result of a lack of available information about the Strategy and its Components.  In particular, representatives of NGOs and P/T governments indicated confusion around separate solicitations for Gs&Cs under the ISHLCD (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

3.2 Design and Delivery

Are the most appropriate and effective means being used to achieve objectives?

This evaluation found a number of challenges in relation to the design and delivery of Diabetes CBP. There appears to be an overall lack of clarity around the roles, responsibilities and processes for program delivery, and no systematic mechanism to support the ongoing monitoring and evaluation of the program.

Clarity on the Roles and Responsibilities in Program Delivery

The evaluation found evidence of a lack of clarity around the roles and responsibilities in delivering programs/initiatives under the ISHLCD, including Diabetes CBP.

There were conflicting points of views in terms of whether or not there is role clarity between the National and the Regional Offices in the delivery of Diabetes CBP.  Moreover, it was noted while respondents emphasized a need to use Regional Offices in delivering the ISHLCD, the roles of the National Office compared to the Regional Offices are not clearly delineated under the ISHLCD in general.  This is consistent with findings from a case study undertaken as part of the Second Implementation Review of the ISHLCD. This case study found challenges with the operational structure in implementing activities under the Strategy, and particularly found a lack of clarity around roles and responsibilities between the National and Regional Offices (Public Health Agency of Canada, 2008c).

At the same time, this evaluation also identified challenges with the delivery of the ISHLCD in the regions.  The regional program representative maintained changes associated with moving from the original CDS to the renewed CDDS under the ISHLCD had some provinces questioning the role of PHAC in delivering some of the Components.  The regional program representative also noted the broader challenge in managing provincial stakeholder relations within the uncertain environment in which Diabetes CBP operates. For example, there is need to explain how some Gs&Cs funding comes without interruption, while other programs, such as Diabetes CBP, are less consistent.

Another challenge relayed by the regional program representative was the apparent confusion among external stakeholders primarily pertaining to the understanding of how different Functional Components of the ISHLCD relate; in particular, how KDED and Surveillance are integrated with CBP.  It was noted that Diabetes CBP staff in the Regional Offices continue to receive requests for information on KDED, Surveillance and Healthy Living from external stakeholders who did not differentiate between Functional Components.  Stakeholders consulted through the Diabetes Policy Review confirmed this finding (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

Consistency in Program Delivery among Regional Offices

Evidence found through this evaluation suggests that both resource constraints and the delay in project funding in many regions have led to a lack of consistency in how Diabetes CBP has been delivered. It was noted that Regional Offices found delays in funding approvals for the solicitation processes for Gs&Cs a major challenge.

As noted by a program representative, the capacity level of staff to deliver the program, including connecting and working with local stakeholders and supporting knowledge transfer, seems to vary significantly among the Regional Offices.  The regional program representative maintained that differences in the availability of human and financial resources among the regions make it difficult to ensure consistency. The allocation of human resources in the regions is based on how much funding each region receives from the ISHLCD, and not on the number of funded projects or level of project funds.  It was noted that consistency can only be achieved if the Regional Offices are given an approval for solicitation of Gs&Cs in a timely manner, and are equipped with the necessary funding and human resources.

Approaches to Solicitation Process for Gs&Cs

The call for proposals used under Diabetes CBP between 2005-06 and 2006-07 varied across the program.  Respondents attributed this to the program’s inability to receive timely funding approval for the solicitation process for Gs&Cs.  All solicitations for Gs&Cs followed Standard Operating Procedures; however, due to time constraints, processes were undertaken within a very short timeframe.

To date, the national and regional solicitation processes for Gs&Cs have used either a targeted or directed approach (i.e., there have been no open solicitations).  Feedback received from several regions indicates the use of a directed solicitation process for Gs&Cs has been primarily a result of limited available funds and tight timelines for the solicitation.  For the 2006-07 solicitation, the only year when all the regions were able to solicit, most regions used a targeted approach, sending out the call for proposals to between 25 and 100 organizations. A few regions used a directed approach, reaching out to between one and five organizations.

Use of Grants

Program representatives indicated, to date, more grants have been used to fund projects compared with contributions. This is for several reasons, including the need to avoid lapsing program funding.  All program representatives agreed on their strong preference for using contributions over grants, as they provide greater control/monitoring of the results, as well as the possibility of providing funding over multiple years (a contribution provides greater flexibility for modification after the project starts).

Clarity around the Solicitation of Gs&Cs and Proposal Review Processes

As noted above, this evaluation found evidence that the process for solicitations of Gs&Cs under Diabetes CBP had not been consistently applied, due to the program’s inability to receive timely funding approval for the solicitation process for Gs&Cs.  In addition, there was evidence the solicitation and proposal review processes for Gs&Cs under Diabetes CBP had not been well communicated to potential funding applicants.

Stakeholders consulted through the Diabetes Policy Review agreed communications from the program regarding solicitations for proposals were insufficient.  Provincial representatives maintained they were not informed of national programs funded under Diabetes CBP and, in some cases, had little information about projects in their provinces that were funded under different funding streams of the ISHLCD.  In addition, both P/T representatives and NGOs emphasized the challenge of project implementation when receiving funding very late in the fiscal year (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

A review of data related to the 2006-07 solicitation process for Gs&Cs indicates a call for proposals in most regions and for the national consultation was sent in late September to mid-October 2006.  However, in some regions, a call letter was not sent until early November, or as late as January 2007.  In most regions, potential applicants were asked to submit their project proposals within 4-6 weeks, while applicants in two regions were given less than three weeks.  Those applying for funds in most regions were then notified the estimated timelines for decisions on their proposals would likely be 12 weeks; this in turn delayed project implementation.

Stakeholders consulted through the Diabetes Policy Review noted a lack of openness and transparency in the proposal review/selection process under Diabetes CBP (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).  The Guide for Applicants for 2006-07 outlined the three-step process used to review the proposals submitted: 1) internal screening by program staff; 2) review by a committee consisting of internal and external reviewers selected on the basis of their experience in diabetes and related fields; and 3) final approval.  However, feedback received from the National and Regional Offices indicated the review process had not always involved an assessment by external reviewers.

For the 2006-07 national solicitation process for Gs&Cs, PHAC national and regional diabetes staff, other disease/issue areas or population groups, and one external reviewer reviewed proposals.  For a targeted solicitation for Gs&Cs in the regions, following the initial screening, a review committee was established to review and rate eligible proposals using a Proposal Review Form. The standard form, or a similar form with a different set of criteria, was used nationally and in the regions (weightings for the assessment criteria were customized by each region, based on their respective priorities stated in a call for proposals).  However, the composition of the review committee varied among the regions.

Gaps and Challenges in the CBP Functional Component

This evaluation found a number of gaps and challenges in the CBP, including challenges with the solicitation process for Gs&Cs.

Ability to Solicit for Gs&Cs

Program representatives agreed the implementation of the ISHLCD, including the CBP, had been hindered by a number of external factors, such as a change in government and the recent Diabetes Policy Review.  It was emphasized that the solicitation process for Gs&Cs under Diabetes CBP had been a major challenge.  In particular, approval of solicitations for Gs&Cs that came late in the fiscal year led the program to use a targeted or directed approach to the solicitation.  As a result, the program has not been able to hold an open solicitation for project proposals, which could have better addressed community-level needs.

Areas of Focus in Funded Projects

To date, the focus of Diabetes CBP has been to fund projects that address health promotion and diabetes prevention among high-risk groups, and, to a lesser extent, prevention of complications for people living with diabetes.  A limited number of projects focused on early detection of diabetes, including screening.

Many stakeholders consulted through the Diabetes Policy Review identified screening and self-management as areas requiring immediate focus.  In particular, providing comprehensive support to the self-management of diabetes was considered a priority in light of the growing pressure on the health care system (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

Integrated Approach to CBP in Healthy Living and Chronic Disease Prevention and Control

As noted in the fully-elaborated RMAF, integration within the CBP Functional Component was expected to take place on a number of levels, such as through targeting common risk factors (e.g., unhealthy eating and physical inactivity).  Stakeholders consulted through the Diabetes Policy Review also supported the integration of diabetes prevention with the prevention of other chronic diseases (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

Progress has been made at the program level on the integration between Diabetes CBP and the Healthy Living Fund, particularly through the new Community-Based Network and the Diabetes and Healthy Living Evaluation Working Group.  As identified in a July 2005 presentation by a regional program representative (Public Health Agency of Canada, 2005b), however, a challenge appears to exist in determining how Diabetes CBP fits with other funding programs.  Stakeholders consulted through the Diabetes Policy Review expressed confusion as to which projects best fit with which funding programs.  Furthermore, several P/T representatives noted the challenge with disease-specific funding as they work towards the integration of chronic disease prevention and management (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

A case study that examined integration within the CBP Functional Component, as part of the Second Implementation Review, found a significant challenge to improving integration within the CBP was the absence of a clear definition of the concept of integration, as well as the operational mechanisms to support it (Public Health Agency of Canada, 2008c). A program representative further noted that for the CBP to evolve into a complete Functional Component under the ISHLCD, other disease specific components (e.g., mental health, cancer, CVD) would need to be added to the current structure.  The program representative also indicated there are opportunities to build synergies with PHAC’s Population Health Fund, which covers a number of areas related to different programs/initiatives within PHAC.

Use of Multi-Year Funding

Both PHAC staff and external stakeholders expressed concern over the inability to use multi-year funding in Diabetes CBP.  Approximately 40% of the 2006-07 funded projects that provided reporting information noted the constraints of single year funding as a key barrier to reaching their target populations.  For example, one project noted its commitment to involving its community partners in the project development process had made it very difficult to meet the project timelines, suggesting multi-year funding would help implement projects that are more meaningful and successful in achieving expected results.  Representatives of NGOs and P/T governments consulted through the Diabetes Policy Review noted that single-year funding, with all its administrative requirements, presented a significant challenge to the continuity and long-term value of projects (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).  The ability for projects to develop and implement strong evaluation plans would also likely improve with multi-year funding.

The Second Implementation Review of the ISHLCD found the lack of provision of multi-year contribution agreements under the Strategy had hampered communities’ ability to deliver on project objectives, which in turn compromised their ability to effectively attain the Strategy’s outputs and outcomes (Public Health Agency of Canada, 2008c). A July 2005 presentation by a regional program representative (Public Health Agency of Canada, 2005b) also argued longer-term funding is required to: allow community-based organizations to continue to build momentum; support focused interventions aimed at at-risk populations and groups who have little infrastructure and resources; and focus on upstream determinants of health and early prevention, rather than primarily targeting risk factors. In addition, program representatives noted multi-year funding is expected to improve the program’s collection of rigorous evaluation data from funded projects, as well as enable greater collaboration.

Dissemination and Other Activities to Support Knowledge Transfer

The findings of this evaluation identify a need to help develop and enhance the capacity of funded projects to evaluate the results of their interventions. Participants at the National/Ontario Information Sharing Meeting in November 2007 (Public Health Agency of Canada, 2007c) identified challenges in ensuring projects focused on evaluation and dissemination of project results. Despite the importance of evaluating community-based programs and sharing the results and experiences to facilitate the transfer of best practices, evaluation was viewed as an area of weakness under the renewed CDS by stakeholders consulted through the Diabetes Policy Review (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).  There was consensus on the need to develop and enhance the evaluation capacity among those who deliver or coordinate diabetes programs/initiatives.

Stakeholders consulted through the Diabetes Policy Review unanimously emphasized the need to support knowledge transfer from community-based programs through information dissemination and support linkages among funded projects (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).  Provincial representatives, health professionals and diabetes educators agreed information about funded projects should be published to allow other programs to learn from, or build on, their experiences.  Also, a number of health professionals stressed the need to build linkages between community-based programs and academic/health research institutions to facilitate knowledge transfer.  These health professionals considered it important to encourage researchers to become involved in community-based programs, as individual projects often lack the capacity to transfer knowledge generated through their community-based activities.

Integration Across Functional and Program Components

As noted by a program representative, one intention of the ISHLCD was to facilitate integration between the CBP and the KDED Functional Components to maximize the Strategy’s investments.  However, full operation of the model based on inter-related components has been significantly challenged by the fact the ISHLCD has not been implemented as planned.

Measuring Outcomes from Funded Projects

This evaluation found there is a disconnect between the existing monitoring and evaluation mechanisms for Diabetes CBP and the Community-Based Programming and Community Capacity Building Functional Component Monitoring and Evaluation Plan.  As currently designed, PERT does not collect sufficient data for a number of outcome areas identified in the Monitoring and Evaluation Plan.  Collection of additional data on the outcomes of funded project activities was attempted through the Early Outcome Evaluation for selected 2006-07 projects; however, this exercise revealed a lack of systematically collected data for some of the key outcome areas for many funded projects.

Key Lessons Learned to Date in Reaching Target Populations

There are a number of key lessons learned from Diabetes CBP with regards to reaching target populations and working with ethno-cultural communities.

All 2006-07 funded projects that provided reporting information identified one or more barriers to reaching target populations.  Figure 2 presents these barriers.

Figure 2: Barriers to Reaching Target Populations (2006-07 Funded Projects)

Funded projects also identified lessons learned or recommendations on reaching target populations, as presented in Figure 3.

Figure 3: Lessons Learned/Recommendations Related to Reaching Target Populations

Practitioners consulted through the Diabetes Policy Review also provided some lessons learned in reaching target populations.  Diabetes educators noted the importance of a comprehensive approach, which involves interventions at multiple settings, such as schools, home/families, workplaces as well as food industries.  It was stressed that public education should take a holistic approach in order to effectively change people’s behaviours (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

A number of 2006-07 funded projects provided lessons learned in working with ethno-cultural communities.  There was recognition among projects of the importance of developing outreach strategies for ethno-cultural communities that build on existing community networks. This helps to ensure potential cultural or linguistic barriers do not limit gathering and sharing of knowledge.  In addition, several projects emphasized the need to adopt a social determinants of health perspective, as opposed to traditional structured educational and social marketing approaches, in reaching these populations.  Projects also stressed the importance of recognizing the overwhelming social and health disparities experienced by these communities, including difficulties in changing behaviours that are determined by socio-economic forces. At the same time, it was noted meaningful changes to address the complexity of serving disadvantaged communities with multiple barriers is a long-term process involving policy and systemic changes.

Researchers, health professionals and diabetes educators who were consulted through the Diabetes Policy Review maintained that existing prevention and health promotion programs were failing to reach the people who need to be reached, namely high-risk populations.  They emphasized the need to rethink the current approach to primary prevention of diabetes (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).

Effectiveness of the Current Monitoring System

This evaluation found evidence that the existing monitoring system under Diabetes CBP was not effective in measuring the outcomes of 2006-07 funded projects.  Project reporting through PERT appears to require significant improvement in methods used and quality assurance of the data.  Moreover, completion of the tool is not mandatory for grants and, as a result, the program cannot monitor results of projects that chose not to complete PERT.

The Monitoring and Evaluation Plan for Community-Based Programming and Community Capacity Building Functional Component, as part of the fully-elaborated RMAF, identified two ongoing data collection tools that were designed to be complementary, namely PERT and the Community Capacity Building Tool (CCBT).  The CCBT, which was designed to capture process-related information about changes in community capacity that occurred throughout the course of a project, was never implemented since it was originally designed to monitor contribution agreements and was not well suited for grants.  Program representatives agreed the requirement to use grants over contributions has hindered the program’s ability to monitor outcomes of funded projects.

A review of 2006-07 PERT and the data submitted by funded projects suggests a need for more comprehensive training of funded projects in completing the tool, and to further clarify the questions.  Duplication in data reported at six months and at the end of project was found for a number of projects, resulting in potential double counting from both reporting periods.

Upon completion of the PERT pilot, a validation of the tool was undertaken, and an improved version was used for 2007-08 funded projects.  An assessment of the improvements to PERT was not undertaken as part of this evaluation.

In addition, the collection of program-level data undertaken as part of this evaluation revealed a lack of ongoing monitoring of the administrative data at both the national and regional levels.  For example, some key corporate information, including the results of solicitations for Gs&Cs and review processes, were not readily accessible for this evaluation. This finding raises concerns about the program’s ability to monitor and provide data on program activities and administrative procedures.

3.3 Success/Progress

Is the Component effective and without unwanted outcomes?

In summary, the implementation of the ISHLCD, including the CBP, has been significantly challenged by a number of factors from within PHAC as well as the broader environment in which it operates.  While the Strategy was announced in October 2005, implementation efforts were impeded as a result of the election call in November 2005 and subsequent change in government.  The delay in obtaining funding approval for the solicitation process for Gs&Cs in 2006-07 resulted in delayed project implementation in many regions as well as at the national level.  However, despite these influences, this evaluation found limited but promising progress in achieving results under Diabetes CBP, as described below.

Success and Progress at the Project Level

Use of Evidence in Planning, Implementing and Evaluating Interventions

Diabetes CBP is designed to support evidence-based interventions aimed at the prevention and management of diabetes.  Information collected through this evaluation suggests most funded projects used some form of evidence in identifying or defining the issues they aimed to address, as well as in planning their approaches and activities.  However, use of evidence to examine project activities and outcomes appears to have been more limited.

A review of project proposals funded in 2006-07 suggests the majority of projects indicated use of evidence in identifying issues they aimed to address.  Several proposals stood out with well-developed rationales describing the existence of the issue and relevance of the proposed initiative; however, approximately a quarter of proposals included limited or no sources of evidence to support the proposed work.  The lack of use of evidence to support projects may be partly attributable to the short timelines projects were given to prepare proposals.

Many funded projects indicated use of evidence to inform the design and implementation of project activities.  Approximately half (or n=20) of 2006-07 proposals referred to studies, reviews or lessons learned from past projects.  Performance monitoring data also revealed many used evidence to inform project design, approaches and concepts.  For example, 20% of reporting projects used evidence collected through a review of literature, existing programs, or environmental scans, and 11% through a needs assessment.  In addition, 60% of projects collected feedback from members of target populations, experts or key stakeholders to improve their approaches.

Use of evidence in evaluating interventions appears to have been less widespread.  Only 20% of projects indicated having developed an evaluation framework, with few identifying the type of evidence used to evaluate their activities.  Projects’ abilities to develop strong evaluation plans may have been influenced by limited capacity and time to prepare and complete the project activities.

As noted in Section 2.2 “Methodological Limitations”, Early Outcome Evaluation reports prepared by 17 projects showed differing approaches among the evaluations undertaken, and a varying degree of rigour in methods.  The methods used by most projects to collect evaluation data raise concerns about data validity.  While some of these limitations may reflect the challenges in developing evaluation strategies midway through project implementation, this points to the need to develop and enhance evaluation capacity of funded projects.

Reaching Target Populations

Overall, a review of 2006-07 data suggests projects generally succeeded in reaching their target populations.  However, numbers of individuals reached varied widely by type of project and the intensity of intervention.

Diabetes CBP is designed to address three priority areas under the renewed CDS: 1) prevention of diabetes among high-risk groups; 2) early detection of type 2 diabetes; and 3) management of type 1 and type 2 diabetes. Many projects aimed to address more than one priority area.  Out of 41 projects funded in 2006-07, 90% aimed to address Priority Area 1; 20% aimed to address Priority Area 2; and 29% focused on Priority 3. Hence, while the Diabetes CBP was designed to benefit populations who are at higher risk for developing type 2 diabetes as well as those living with diabetes, the majority of funded projects aimed to address prevention among high-risk groups, and as a result, the program’s reach to people living with diabetes was limited.

In terms of target groups, approximately 68% (or n=28) of funded projects had objectives with a focus on one or more specific group, while 13 had a more general focus.^{Footnote 7} Of these, ten projects focused their activities on more than one target group.  As illustrated in Figure 4,^{Footnote 8} “youth, families with children and youth, or parents” (n=10) and “persons who are physically inactive” (n=8) were most frequently targeted by the projects.^{Footnote 9}

Figure 4: Frequency of Target Groups Selected

A review of performance monitoring data for 2006-07 funded projects indicated most projects reached: individuals/communities

^{Footnote 10} facing specific risks, followed by practitioners, professionals and other service providers.  Policy-makers and the general public were less frequently targeted as shown in Figure 5 below.^{Footnote 11}

Figure 5: Target Population Reached (2006-07 Funded Projects)

Figure 6 presents an estimated number of individuals in each of these four target groups that 34 projects

^{Footnote 12} reported reaching.^{Footnote 13} The number includes individuals who may not have actually participated in project activities (e.g., individuals to which a brochure was mailed, Website hits) but were indirectly reached.^{Footnote 14}

Figure 6: Estimated Number of Individuals Reached (2006-07 Funded Projects)

The number of individuals reached varied greatly depending on the nature of the project’s objectives.  High-intensity educational or risk-modification programs generally included fewer people; for example, a project aimed at preventing the development of diabetes among people with pre-diabetes or a history of gestational diabetes through educational and fitness programs reported reaching 91 individuals with pre-diabetes and 50 women with a history of gestational diabetes.  On the other hand, projects focusing on awareness and sensitization were in contact with more people but with less intensity; for example, a project aimed at increasing awareness of diabetes among francophone seniors in New Brunswick reported reaching over 1,200 seniors through health forums across the province.

Educational, Awareness or Outreach Activities

A review of 2006-07 reporting data indicates almost all projects held activities designed to raise awareness among, or educate, target populations on issues related to the prevention or management of diabetes and other chronic diseases, and most also developed and disseminated materials/resources.

All of the 2006-07 funded projects providing reporting data indicated having held education, awareness or outreach activities.  Figure 7 presents the average number of activities per project by type of activity.^{Footnote 15}

Figure 7: Average Number of Education, Awareness and/or Outreach Activities Held (2006-07 Funded Projects)

Projects delivered activities in a number of settings, with the majority (73%, or n=24) indicating they had held activities in multiple settings.

^{Footnote 16} As shown in Figure 8, these awareness/outreach activities were most frequently held among local community groups (88%, or n=29 reporting projects), at a recreation/sport facility (88%, or n=29) and in a health setting (61%, or n=20). The “other” settings reported by nine projects include, among others, family settings, a restaurant, a hotel, online, and through a publication.^{Footnote 17}

Figure 8: Settings for Education, Awareness and/or Outreach Activities (2006-07 Funded Projects)

Most projects developed and disseminated materials of various types for their target populations to increase access to diabetes-related resources.  All but one project reported having developed education, awareness and/or outreach products over the project year.

^{Footnote 18} Table 3 presents the number of materials distributed (total and average) by type of product.^{Footnote 19}

As shown in Table 3, there was a significantly wide range in terms of the number of products distributed across all product categories.  For example, a project aimed at preventing, detecting and managing diabetes in DaimlerChrysler Canada employees, retirees, and family members distributed over 43,000 copies of 28 different brochures and information sheets, and 75,000 copies of three communiqués/newsletters.  Small numbers were generally reported for articles, press releases and advertisements that were published in print media or on a website.

It should be noted that producing and disseminating resources, or holding programs or activities, does not automatically result in improved access to the resources for the target population. Available data do not provide information on uptake or use of resources in the target population. There was little evidence provided to ascertain whether information made available to target groups was actually received, comprehended and applied by recipients.

Partnership Development and Collaboration

Diabetes CBP is designed to promote participation by members of the target group and other key stakeholders, as well as intersectoral collaboration in project development and implementation.  A review of data on projects funded in 2006-07 suggests, overall, partnerships were a key feature of funded projects.

All of the 35 projects providing reporting information indicated having formed partnerships over the project period, identifying a total of 426 partners or an average of 12.2 partners per project.  Approximately 85% reported having formed new partnerships as a result of the projects, with a total of 203 new partnerships, or an average of 6.8 new partners per project.

Figure 9 provides a breakdown of these new partners by type of organizations.^{Footnote 20}

Figure 9: Number of New Partners by Type of Organization (2006-07 Funded Projects)

As shown in Figure 10, over half of the new partners were from the health sector, followed by education (11%, or n=22) and social services (10%, or n=21).

^{Footnote 21}

Figure 10: Number of New Partners by Primary Area of Focus (2006-07 Funded Projects)

A number of projects reported various positive outcomes resulting from partnerships formed for their projects.  Examples of these results included: acquisition of monetary and non-monetary resources; gaining access to target audiences; gaining assistance and advice with project administration and implementation; acquisition of useful knowledge and information; involvement of specialists from partner organizations; and expansion of contacts with other organizations.

Data collected through this evaluation provided some evidence demonstrating success with regards to the program objective of mobilizing collaboration to integrate systems of prevention, detection and management of diabetes. Several projects reported some early success in supporting the development of a coordinated systems approach to the prevention and management of diabetes and other chronic diseases. For example, participation on a national advisory committee achieved increased knowledge, engagement and dialogue among public and primary health care networks regarding a systems approach to strengthening chronic disease prevention and management. In reality, systems level mobilization would be difficult to achieve in a one year period.

Impacts on the Awareness and Knowledge among High-Risk Populations

Neither PERT reporting nor the Early Outcome Evaluation data could be used to assess the success in increasing awareness and knowledge among high-risk populations. Current reporting tools were not designed to assess this level of outcome. Most funded projects reported their activities helped increase awareness and knowledge about diabetes, its risk factors and prevention, as well as other relevant issues; however, these claims were often not substantiated by evidence.

Approximately 31% (or n=11) of funded projects providing reporting information noted changes had occurred in their target populations including: increased awareness and knowledge (reported by eight projects); changes in attitudes and behaviours (three); improved skills in managing diabetes (one); and increased access to programs and services (one).  However, only five projects presented systematically collected evidence for changes that had occurred.

Most (n=13) of the 17 projects participating in the Early Outcome Evaluation indicated projects had helped increase participants’ awareness or knowledge about diabetes, its risk factors and prevention, or changed participants’ perception or attitudes about diabetes.  As well, seven projects reported that participants indicated changes in their thinking, attitudes, and/or understanding of diabetes as a result of the projects.  For example, a project aimed at preventing diabetes among women with a history of gestational diabetes noted participating women were significantly more likely to see diabetes as a serious health condition following participation in the program.  However, only half of projects provided systematically collected evidence to substantiate these changes.

Impacts on Public Policy

A review of data submitted from funded projects indicates limited but tangible success had been made in attempting or intending to influence policy.  This was mostly achieved at the local level or with regional health authorities.

Approximately 41% (n=13) of 2006-07 funded projects providing reporting information indicated that influencing policy development, policy implementation, or support for existing policies was an intended outcome of their projects.  Nine projects indicated targeting policy-makers, and, of these, seven reported having actually reached this target groupFootnote 22 through activities designed to influence policy, such as town halls and consultations.  However, these included some projects that indirectly intended to influence policy, for example, through demonstrating a gap in programming related to diabetes prevention.  Others aimed at more directly influencing policy; for example, one project actively built partnerships with local municipalities in an effort to influence land use and planning policies to ensure access to opportunities for physical activity and healthy eating.

Of the 13 projects focused on policy: 12 developed a working relationship with a government or community representative who could provide access to the policy process; 11 held meetings with policy-makers such as school boards, city councils, public health authorities, or community organizations; five participated in a policy development process with policy-makers; four presented briefs or position papers (e.g., to decision-makers); and three prepared and distributed a policy research report.

Many of the 13 projects provided information demonstrating success of project activities in influencing policy, which primarily focused on municipal, local or organizational policies.

Impacts on the Capacity to Apply Best Practices and Clinical Practice Guidelines

This evaluation was unable to collect sufficient data to assess the extent to which funded projects helped improve the capacity of the health care system to apply best practices and clinical practice guidelines related to diabetes screening, education and counseling.  Questions in the PERT do not ask for information about the training of health professionals or service providers in best practices and clinical practice guidelines.

Engagement of Key Stakeholders in Priority-Setting for CBP

This evaluation found stakeholder involvement in priority-settings for solicitation of Gs&Cs under Diabetes CBP has been very limited, with the exception of P/T governments.  To date, there is no formal mechanism to involve a broader group of stakeholders in the priority-setting process.

The priorities set for national projects under Diabetes CBP were identified through consultations with P/T governments, and are based on the priorities identified through the 2001–2003 national diabetes stakeholder consultations.  Formal consultation with P/T governments is undertaken on an ongoing basis through the Public Health Network, a federal-P/T process designed to address issues around infectious and chronic diseases; NGOs are also informally engaged.

In the Regional Offices, priorities for solicitation of Gs&Cs are set on an individual basis by each region (for each province/territory) to avoid duplication and ensure federal funding for community-based projects does not conflict with P/T activities.  In all regions, the priority-setting process involves consultation with P/T health departments/ministries.  Two regions have also involved representatives of the First Nations and Inuit Health Branch (FNIHB) of Health Canada and NGOs in the process; one has also engaged diabetes educators, health authorities, people living with diabetes and universities.  In five out of seven regions, there has been little involvement of non-governmental stakeholders in the priority-setting processes as the delay in solicitations of Gs&Cs resulted in limited time to engage NGOs.

Stakeholder Engagement in Partnership Development and Collaboration at the Program Level

Partnership development and collaboration is one of the three areas of action under the CBP Functional Component.  However, this evaluation found that for Diabetes CBP, at the overall program level, little collaboration has occurred and few partnerships have been developed with stakeholders external to PHAC.

The information collected and reviewed through this evaluation suggests collaboration has occurred within PHAC as well as across Functional and Program Components of the ISHLCD.  However, as indicated by a program representative, partnership development and collaboration with external stakeholders under the CBP has been limited to two activities: 1) formal consultation with P/T governments through the Public Health Network; and, 2) informal engagement of NGOs on the solicitation process for Gs&Cs. Stakeholders consulted through the Diabetes Policy Review indicated that the Public Health Network did not have diabetes NGOs represented on its Expert and Task Groups and that, as a result, it did not offer them a venue for consultation (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b). In addition, representatives from the Canadian Institutes of Health Research (CIHR), provincial governments and national NGOs maintained there is a lack of understanding by all diabetes stakeholders of the overall activities under the ISHLCD, including the CBP, largely as a result of the lack of regular stakeholder communications under the Strategy.

At the regional level, there appears to be a notable difference across Regional Offices in the level of partnership development and collaboration activities, which has largely been attributed to differences in availability of human and financial resources.  For example, Alberta, one of the regions identified as having sufficient resources, has formed a number of partnerships, including one with an academic/research institute aimed to enhance knowledge transfer and exchange.  A regional program representative noted that most regions were not equipped with sufficient resources to engage in activities related to partnership development and collaboration with other players in the field.

Dissemination of Knowledge Generated from CBP

Knowledge dissemination by PHAC is another one of the three action areas under CBP. Analysis, synthesis and dissemination of knowledge from communities, surveillance systems, evaluation and research are planned activities that support knowledge transfer.  This evaluation found that little has been undertaken to disseminate knowledge generated from funded projects at the program level.  The program’s expected activities have not been clearly defined for this action area, and no specific mechanism has been identified for the dissemination of knowledge generated from funded projects under Diabetes CBP.

At the time of project completion, 71% of the 2006-07 funded projects had either produced or nearly completed final evaluation reports, and another 14% indicated having initiated the report development.  Almost all funded projects indicated having disseminated the results and lessons learned from their projects to one or more audiences, most frequently being partners and collaborators (reported by 91% of the projects).

At the program level, information about funded projects or knowledge generated from their activities has not been made available to stakeholders who may benefit from gaining new knowledge. Plans are underway to develop and make available to the public a report containing lessons learned from projects funded under Diabetes CBP in 2006-07; this report will be available in fiscal year 2009-10.  National and Regional Offices may not have sufficient capacity and expertise, or adequate resources, to support knowledge transfer.

Stakeholders consulted through the Diabetes Policy Review noted that lack of access to information about existing community-based programs has resulted in duplication of work and lost opportunities for collaboration and learning from the experiences of existing initiatives (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b). Provincial representatives, health professionals and diabetes educators all emphasized that information on funded community-based projects (including project activities and lessons learned) should be made public to facilitate the transfer of best practices.

This evaluation found a need to develop a common approach to distribute project information (e.g., the name of the organization, a description of project activities) funded under Diabetes CBP.  A centralized mechanism to support the dissemination of evaluation findings undertaken by individual projects is also required.  In addition, there is a need to clarify the types of information projects are generating, what audiences would benefit from this information, and how it should be disseminated.

Supporting the Uptake of Surveillance Data and the Sharing of Knowledge Among Stakeholders

CBP is designed to support the uptake of surveillance data among community-based programs.  This evaluation found evidence of some success in this area at the national level.  As noted by a program representative, staff in the National Office collaborate with the Surveillance team in developing reports that are made available to the public (e.g., National Diabetes Surveillance System Highlights Report).  These reports are designed to be published annually; however, there has been a delay in synthesizing data at the national level and, therefore, reports have not been released annually. Also, CBP within the National Office works with key national NGOs and CIHR to develop an annual National Diabetes Fact Sheet, which is designed to provide diabetes-related information, including data from the NDSS, in a format that supports uptake among the broader audiences. Most regional offices do not have sufficient resources or expertise to support the uptake of surveillance data from resources such as the NDSS.

Unintended Positive or Negative Impacts of Diabetes CBP

Largely due to the lack of both data on the results of funded projects beyond a single fiscal year and the lack of relevant questions in PERT, no systematically collected evidence was available to assess positive or negative impacts of Diabetes CBP.^{Footnote 23}