ARCHIVED - Federal Initiative to Address HIV/AIDS in Canada Implementation Evaluation Report

III. Findings

This section addresses the following six evaluation questions:

A. Relevance

Q1: To what extent is there a continued need for federal involvement in HIV/AIDS issues in Canada?

In 2004, the Federal Initiative emerged from the CSHA, using evidence from research and surveillance studies, population needs assessments (federal inmates and First Nations and Inuit), public opinion surveys, and stakeholder advice.

The emerging focus for the federal government for the period 2004-2008 has been to organize, align, and direct federal efforts and resources in partnership with other stakeholders and to address the five Federal Initiative objectives. Another shift has been to move from putting in place the programs and policies to respond to HIV/AIDS towards developing the science to better understand the disease and design more effective interventions.

HIV/AIDS Epidemiology Updates and Recent Research Findings

Three years after the launch of the Federal Initiative, epidemiological studies and surveillance and research do not show significant changes in the HIV/AIDS epidemic.

HIV/AIDS is still considered a global priority due to the continuing high rates of transmission of the infection, the adaptive capacity of the retrovirus, the absence of effective preventive vaccines, the absence of a cure, the economic cost to society, and impact on key populations.

More Canadians are now living with HIV – an estimated 58,000 at the end of 2005 compared with 50,000 at the end of 2002. PHAC has estimated 2,300 to 4,500 new HIV infections occurred in Canada in 2005 compared with 2,100 to 4,000 in 2002 (D. Boulos, 2005). In 2005, PHAC estimated that approximately 27% of the total number of people living with HIV were undiagnosed and unaware of their condition. This suggests that an increased focus on national interventions to increase awareness of the risk of HIV transmission and improve access to, and use of, HIV testing is necessary.

Recent studies show new trends among vulnerable populations. For example, younger infected people who take drugs are less likely to use antiretroviral therapy (M. Rush, et al.). In addition, the stigma associated with HIV continues to set it apart from most other diseases in Canada and abroad. Public opinion studies undertaken by PHAC and HC (Ekos Research Associates, HIV/AIDS Attitudinal Tracking Surveys) in 2003 and 2006 provide evidence that intolerance and stigma are still associated with HIV/AIDS. Changes in social perception and behaviour require effective mechanisms to disseminate information through social marketing actions and awareness campaigns.

Relevance of Federal Initiative Partners’ Involvement in HIV/AIDS in Canada and Abroad

The roles that partners play in the Federal Initiative are multifaceted. Partners contribute to the following activities:

• prevent Canadians from being exposed to the virus in a changing environment;
• promote healthy behaviour;
• increase quality of life of people living with HIV;
• provide health services to specific populations;
• increase knowledge to improve access to treatments, interventions, and healthy practices; and
• fulfill specific global commitments.

A review of the federal partner activities confirmed that they continue to be relevant to the federal role. Exhibit III-1 illustrates the strong correlation between the Federal Initiative’s long-term outcomes and the partners’ goals and mandates.

Although the findings confirm the continued relevance of the federal government’s role in addressing HIV/AIDS in Canada, recent changes to government priorities will have an impact on the Federal Initiative. For example, the establishment of PHAC (2004) and the new focus on public health within the CSC (2007) suggest that a review of these partners’ functions through a public health lens would be useful to align their HIV/AIDS approaches with their new public health role.

Conclusion

HIV is a public health issue of national concern. Exposure to HIV is preventable, yet the number of infected Canadians continues to climb. The Federal Initiative’s relevance stems from the need for a nationally coordinated effort that protects Canadians from contracting the HIV virus. The role of the Federal Initiative is to lead activities in social marketing; develop evidence-based policy; identify best practices; conduct epidemiology, surveillance, and laboratory studies; fund research; and establish appropriate interventions for target populations.

The Federal Initiative will need to be able to continuously adapt to evolving epidemiological trends and to political and environmental pressures. This evaluation found sufficient evidence to confirm the ongoing need for Federal involvement in HIV/AIDS issues in Canada.

Exhibit III-1 – Federal Initiative Outcomes and Partners’ Responsibilities

Federal Initiative Long-term Outcomes

1. prevent the acquisition and transmission of new infections;
2. slow the progression of the disease and improve quality of life;
3. contribute to the global effort to reduce the spread of HIV/AIDS and mitigate its impact; and
4. reduce the social and economic cost of HIV/AIDS to Canadians.

PHAC is responsible for promoting and protecting the health of Canadians through leadership, partnership, innovation, and action in public health.

The Centre for Communicable Diseases and Infection Control (CCDIC/PHAC) is responsible for decreasing the transmission of infectious diseases, such as HIV/AIDS, and improving the health status of those infected. Regional Offices are responsible for carrying out PHAC's mandate through activities, such as program delivery, research and knowledge development, policy analysis and development, community capacity building, and public and professional education. PHAC Regional Offices also administer the AIDS Community Action Program (ACAP).

HC's goal is for Canada to be among the countries with the healthiest people in the world. HC’s First Nations and Inuit Health Branch (FNIHB) works closely with First Nations on-reserve and some Inuit communities to improve health outcomes; ensure the availability of, or access to, quality health services; and support greater control of the health system by First Nations on-reserve and some Inuit communities.

The department’s International Affairs Directorate initiates, coordinates, and monitors departmental policies, strategies, and activities on the international stage. It provides advice on the department's strategic approach to international affairs; ensures the department's international activities are internally coherent and consistent with government-wide policies; and recommends departmental representation at international meetings.

The CSC, as part of the criminal justice system and respecting the rule of law, contributes to public safety by actively encouraging and assisting offenders to become law-abiding citizens, while exercising reasonable, safe, secure, and humane control. The CSC provides health services, including those related to the prevention, care, and treatment of HIV/AIDS, to offenders sentenced to imprisonment of two years or more. It offers inmates HIV testing, counselling on test results, education on risk reduction, and medical treatment for HIV-infected inmates.

The CIHR was created in 2000 to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.

B. Design and Delivery

The RMAF identifies the following changes that distinguish the Federal Initiative from its predecessor:

• eight priorities (see Exhibit I-1);
• an approach by target population;
• new funding, allocated incrementally over five years; and
• implementation of means to support and coordinate the Federal Initiative: governance structure; performance management and measurement system.

Q2: Did the planned funding by areas of action address priorities and the target populations?

Funding Allocation and Distribution by Area of Action

Funding allocations are delivered through the Federal Initiative’s five areas of action. Exhibit III-2 – Federal Initiative Funding by Areas of Action – illustrates the relative allocations in 2005.

Annex I-2 summarizes the nature of the 13 programs and 3 governance components of the Federal Initiative, their allocation across the participating departments and agencies, and linkages to the five areas of action.

Exhibit III-2

During the first three years of a total budget of $157.6 million, $118.6 million was spent to ensure and enhance the ongoing implementation of activities established by the CSHA; $39 million was invested in the implementation of new activities and policy directions.

Annex I-4 – Federal Initiative Allocations by Areas of Action – shows that funds from fiscal year 2004-2007 were allocated and fully spent. The actual expenditures by area of action are not available because tracking codes were not implemented. In FY 2008-2009, an enhanced financial tracking system is being implemented to correct this gap. PHAC, HC, CIHR and the CSC are using Federal Initiative financial data collection templates developed by PHAC Corporate Finance, which will provide consistent financial details for future reporting.

Funding Allocation and Distribution by Priorities

Activities delivered directly by the federal government and those delivered indirectly are the two main strategies underlying the Federal Initiative delivery model. The following is an overview of the analysis provided in Annexes I-2 and I-3.

Activities delivered indirectly are carried out through Gs&Cs programs. The role of the federal government is to fund national or local organizations or researchers (mainly through academic institutions) to undertake activities related to Federal Initiatives priorities. The following programs fall under this category:

• national Gs&Cs (HIV/AIDS Division - PHAC);
• regional Gs&Cs (ACAP – PHAC Regions);
• research grants and awards funded by CIHR;
• First Nations and Inuit on-reserve health programs (HC); and
• HIV/AIDS Global Engagement Grants (HC).

Exhibit III-3 – Federal Initiative Grants and Contributions Programs – shows that the Federal Initiative increased investments dedicated exclusively to Gs&Cs from 2003-2006 by approximately 40%. In 2006, 62% ($39 million) of the Federal Initiative budget was invested in Gs&Cs programs. The CIHR research investment increased from $10 million to $15.7 million (60%). Funding for PHAC regional Gs&Cs managed by ACAP increased by 42%, from $7.4 million to $10.5 million. Funding for PHAC national Gs&Cs, managed by the HIV/AIDS Division, increased by 11% from $9.8 million to $10.9 million. These funds were fully allocated during the study period.

Exhibit III-3

This evaluation confirms that funding was allocated to activities as planned and that the Federal Initiative strengthened and sustained indirect investments.

Activities Delivered Directly

The following activities were delivered directly through O&M funding:

• national surveillance and risk assessment (Surveillance and Risk Assessment Division, PHAC);
• HIV and STI health services programs for specific populations (First Nations and Inuit Health Branch, HC, and federal offenders, CSC);
• national HIV laboratory reference services and HIV strain resistance studies (National HIV Laboratories, PHAC);
• national HIV and STI guidelines (HIV/AIDS Division, Community Acquired Infections Division – STIs, PHAC);
• international activities and global knowledge transfer (International Affairs Directorate, HC, and Surveillance and Risk Assessment Division and National HIV Laboratories, PHAC); and
• population-specific status reports, awareness and social marketing programs, support for co-ordinating and advisory bodies, and reporting and evaluation (HIV/AIDS Division, PHAC).

Exhibit III-4

HC's corporate evaluation services supported evaluation activities. (This function was transferred to PHAC Corporate Evaluation in 2007). The Director General’s office of the Centre for Infectious Diseases and Prevention Control facilitated communication for Federal Initiative mandatory reports.

Exhibit III-4 highlights the following findings:

1. Except for two small funding allocations assigned to support mandatory reports and corporate evaluation, all programs have seen their funds increased. The budget for health services for First Nations and Inuit rose from $0.2 million to $1.2 million a year, and the health services budget for federal offenders increased from 0.6 million to 2.4 million annually. Both programs received significant increases in 2004 and 2005.
2. The budget allocated to the global engagement component activities increased from $0.3 million to $0.8 million a year. This program was responsible for coordinating the federal government’s participation in the International AIDS Conference in Toronto in 2006 by hosting an international policy dialogue on HIV and prisons; developing new regional alliances; influencing global policy through United Nations Boards, Commissions, and the United Nations General Assembly Special Session on HIV and AIDS;and increasing coherence across the federal government’s global HIV/AIDS activities.
3. The HIV/AIDS Division has seen its salary and O&M budget increased by approximately 80%, from $4.4 million to $7.9 million. This RC includes governance and accountability, as well as support for coordinating and advisory bodies (National Aboriginal Council on HIV/AIDS, Leading Together Championing Committee), communications, policy and program development, and social marketing programs. The new priorities for funds allocated to the governance and accountability activities have changed each year as the Federal Initiative has matured. In 2004 the priority was to focus on coordinating/aligning federal programs. In 2005 more effective horizontal management was the driver and the focus on undertaking the evaluation process began in 2006.
4. The national surveillance and risk assessment program has seen its funds increase by approximately 78%, from $2.3 million to $4.1 million a year. Additional funds were used to develop and implement second-generation surveillance tools enabling multi-factorial analyses, which integrate behaviours with socioeconomic and epi data. This program works in close collaboration with the Community Acquired Infections Division (CAID), which is funded by the Federal Initiative for work on STIs. CAID received a budget increase of 43%, from $0.7 million to $1.0 million annually. The funding of both programs correlated with the 2005 planned activities.
5. PHAC’s National HIV Laboratories have seen their funds increased by approximately 30%, from $2.4 million to $3.1 million a year. The laboratories offer high technology services and expertise to provincial and territorial HIV laboratories. The aim is to strengthen and maintain accurate screening for HIV and act as a national reference centre for HIV, providing standard guidelines on laboratory sciences.

This study confirms that the new and ongoing funding has been directed to Federal Initiative priorities.

Target Populations

The Federal Initiative’s eight target populations have been reached through direct and indirect activity. Programs working with federal inmates, as well as the First Nations and Inuit, are reaching target populations directly. Programs working on awareness campaigns are not only reaching the target population, but are also influencing the general population through posters, conferences, and educational material. Information gathered from PHAC’s Gs&Cs Project Evaluation and Reporting Tool (PERT), which collects information on common outputs and outcomes across programs, confirms the reach of the PHAC national and regional programs to the target populations.

Research and surveillance programs are considered indirect activity in using research data to identify needs, best practices, models, and approaches.

Exhibit III-5 summarizes the various means Federal Initiative programs reach target populations.

It is a challenge to measure the target populations reached by programs. Inconsistent program reporting has been an issue, for example, an individual can be part of several different populations at the same time. Measuring the total number of individuals reached (denominator) is currently a methodological problem that will impact on the summative evaluation.

Performance measurement systems are being adapted, but the lack of benchmarks by which to compare progress has impeded the ability to measure program advances.

Annex I-7 – Design Summary by Areas of Actions (“Population” Column) – links the information collected by this evaluation from programs with Exhibit III-5.

Conclusions

This evaluation found that there is a strong correlation among priorities, plans, funding allocations, and expenditures. The majority of programs received budget increases. Allocations to front-line and research programs (Gs&Cs) increased by 40% from 2003 to 2007. In 2006-2007, Gs&Cs programs received a total of $39 million or 62% of the total Federal Initiative budget.

The Federal Initiative also funded activities delivered directly by federal partners. Although the partners spent the funds on the planned activities, the weaknesses in the financial reporting of funded activities affected the ability to link spending with key outputs. The budget is allocated by areas of action. In cases where financial coding was developed to report expenditures, these codes were not implemented adequately. The development of standard practices to improve the quality of financial reporting, including standard operating procedures for coding, reporting, and measurement, will make it possible to align expenditures with key outputs.

The population-specific approach is well implemented by the programs. Measuring the population profile continues to be a challenge. This approach will need to be refined to better track progress towards addressing the target population.

This evaluation found that Federal Initiative planned funding by areas of action addressed the priorities set out in the budget allocations for 2004-2007. Target populations were addressed by the overall Federal Initiative delivery strategy.

Q3: To what extent have Federal Initiative partners implemented new activities?

To answer this question, activities described in the Federal Initiative program documents and DPRs were examined. Annex I-3 (Implementation Status of Federal Initiative New Investments) itemizes each activity, the analysis; and the degree of implementation achieved (measured on a scale - Full Implementation, Advanced Implementation, Implementation In Progress, or Not Implemented, as described in the methodology). The following is a summary of the findings:

It was observed that the outputs from research and surveillance programs were consistent with the goals of the Federal Initiative. New activities were fully implemented as planned, in part because Federal Initiative funding made it possible to implement plans that were in an advanced stage of development before the new funds were received.

 

Most community programs are at an advanced stage of implementation. Community programs are delivered through Gs&Cs and function on a multi-year cycle. It takes three to five years to change program directions. PHAC ACAP and national Gs&Cs programs were all aligned with the Federal Initiative by 2007. Exhibit III-6 demonstrates the multi-year cycle and the implementation over three years.

In 2005-2006, PHAC invested $21 million through Gs&Cs funding to community organizations to support HIV/AIDS prevention and promote access to diagnosis, care, treatment, and support for those affected. Projects funded at the national (52) and regional (148) levels are intended to improve knowledge and awareness of HIV/AIDS and strengthened community and public health capacity to respond to the disease.

Activities for the International AIDS Conference in Toronto were fully implemented and key outputs were provided. This high profile event, with well-structured plans, a short-life cycle, senior management support, and assigned human resources, was an example of an activity that was appropriately resourced for successful implementation.

National and regional Federal Initiative Gs&Cs programs used PHAC’s standardized PERT to collect data on program performance. ACAP added specific questions to the tool to collect project outcome information.

An initial study of performance information (PMF Inventory, 2006) found that the Federal Initiative had developed over 200 program indicators in response to various reporting requirements. The study recommended a revision and update of the RMAF to align program objectives with the Federal Initiative’s logic model key outputs and outcomes, and to confirm the core indicators that will support future reporting and evaluation needs.

The theoretical logic model has been partially implemented. Activities have been aligned with areas of action; however, key outputs need to be identified and fully described under the areas of action. Validation of data sources, timelines for data collection, and a performance measurement implementation plan are critical gaps.

The global engagement component has well-defined activities. An IAD report identified activities and key outputs and outcomes. Common to other RCs, there is no correlation of budget allocations with key outputs under this area of action.

Federal Initiative managers confirmed program strengths and identified weaknesses across the five areas of action. Strengths included extensive cooperation between federal partners and the success of activities based on proven public health interventions. Weaknesses included a shortage of human resources. Programs that encountered implementation difficulties had well-defined plans and activities. The difficulties had more to do with cumbersome processes for the recruitment of specialized human resources and with the high level of mobility of specialized human resources. The Public Service must compete for specialized resources, such as doctors, nurses, and analysts. This competition has impeded the implementation of activities within established timeframes, particularly in FNIHB and the CSC where health professionals deliver Federal Initiative programs.

Conclusions

Most new activities are well established. Surveillance and research key outputs are easier to distinguish and measure than community outputs. The tools to measure program and project outcomes are not yet implemented, which impacts the ability to speak to outcomes. The challenge is further complicated due to the complexity of managing horizontal initiatives within the federal government context. The challenge for the four partners is to develop a common, shared system for mandatory reporting, management, and control, in an environment where each partner has also to answer to a vertical (departmental) system of management and control. Developing a single reporting and management system is a priority. It will strengthen governance and accountability and support the development of the building blocks for the summative evaluation.

The Federal Initiative’s key outputs (Annex I-7 – Design Summary by Areas of Action) were not defined consistently across all programs. Processes to develop a common understanding and increase capacity for planning by measurable objectives will enhance both vertical and horizontal reporting processes.

Implementation of the recommendations of the PMF study (2006) as a priority for the RCC will build capacity for performance measurement and lead to full logic model implementation.

Q4: To what extent have the governance and performance measurement strategy been implemented?

Importance of Good Governance

Governance is defined as the interactions among structures, processes, and traditions that determine how power and responsibilities are exercised, how decisions are taken, and how citizens or other stakeholders have their say (J. Graham, 2003).

Governance is the expectation that the essential conditions (internal coherence, corporate discipline, and alignment to outcomes) are in place for providing effective strategic direction and support to the Minister and Parliament and delivering results. Humphries (2003) compares the setting in place of good governance of an organization to the construction of a house – if you miss one part or build out of order, the results will be inefficient, unsafe, and likely unusable. The Auditor General’s (OAG) 2005 report, Managing Horizontal Initiatives, stipulates that governance issues are critical. The social return on resources devoted to horizontal programs is a function of the effectiveness of the governance and decision-making processes that are put in place.

The main findings related to governance implementation can be found in Annex I-5 – Performance Management Framework Study Results; and Annex I-6 – Governance Evaluation Results.

Governance Body

The Federal Initiative has a centralized management framework. PHAC has the overall responsibility for program coordination, implementation, and reporting. The four participating departments and agencies are individually accountable for decision-making and delivery of their respective programs. Governance of the Federal Initiative is dispersed across the 11 RCs, which carry out 16 types of activity.

The RCC was established in March 2006 as a governance body with the central role of managing the Federal Initiative. Each RC assigned managers to the RCC. Program development and implementation are formed by the plans and priorities set out in the RMAF. The RCC established the AWG to gather information and make recommendations for reporting, monitoring, and evaluation.

Application of the governance indicators (Annex I-6) found that the RCC reflects best practices for horizontal initiatives because decisions are made through democratic representation and a coordinated approach. The role of the RCC is to promote policy and program coherence among the participating departments and agencies, and to maximize the use of available resources. However, findings indicate that the RCC has not yet reached its full potential. RCC meetings were to occur three times a year. In 2006, the RCC met twice and only once in 2007. The meeting documentation indicates a lack of focus on shared objectives, which may have inhibited the opportunities to review challenges and risks, and evolve plans, strategies, and timelines.

The RCC could be more aware of the overall common risks of its management plan. The Risk-based Audit Framework [RBAF 2004] lists possible barriers to successful program implementation. No formal risk management reports were submitted, although a risk assessment of November 2005 was found. Without effective communication of risk, the Federal Initiative governance body is hindered in its ability to mitigate risks.

The following issues have arisen in 2007-2008 that underscore the importance of the RCC’s role and responsibilities:

• Federal Initiative funding was reduced from $84.4 million to $72.6 million with the biggest impact on PHAC RCs;
• The commitment to report annually on World AIDS Day has not been met since 2005; and
• Non-governmental stakeholders (identified as key to Federal Initiative success) have raised concerns about the lack of timely information concerning the status of the Federal Initiative.

This evaluation supports the findings about governance in the PMF report (2006), which concluded:

... Since the PMF is still under development, there is some confusion regarding roles and responsibilities with respect to performance reporting. In addition, performance information itself does not seem to be well integrated with management decision-making processes, perhaps again due to the fact that the PMF is under development.

Performance Management Framework (PMF)/System Implementation

Some programs are more advanced than others in implementing performance measurement systems. Only 4 of the 13 programs have developed a systematic performance measurement system.

• PHAC national and regional Gs&Cs programs are using Project Evaluation Reporting Tool (PERT) to monitor projects. PERT was tested through regional and national pilot projects to align HIV/AIDS project activities with indicators.
• CIHR has aligned research activities with its strategic objectives. CIHR has developed performance indicators to document outputs and impacts of funded research and program success against Federal Initiative objectives. The indicators have not been evaluated in the context of Federal Initiative reporting needs.
• FNIHB (Health Services) has developed a performance measurement framework. This system gathers information from the regions and forwards it to the national office. This data was not available for this assessment and the PMF has not been evaluated from the point of view of Federal Initiative reporting needs.

In general, reporting is done on an as-needed basis, and programs submit descriptions of activities instead of performance data to report on areas of action. The data available is insufficient to demonstrate framework coherence. The theoretical logic model is not implemented at the operational level, and the program’s key outputs and activities are not fully aligned with Federal Initiative outputs. All the elements are present, but not in the required format to align costs with outputs or outcomes.

All reporting is compromised by this weakness, a finding coherent with the report on the PMF (2006), which concluded that indicators needed to be refined and targets prioritized.

In terms of indicators, a good deal of work needs to be done at this level in order to reduce duplication, streamline the indicators and find the critical few. In addition, further review of indicators should be undertaken to ensure more coherence among the indicators themselves; to strengthen the quality of the indicator; and to make the indicators more outcome-based or more informative. A population lens should be included. There is also a need to align the indicators with the Gs&Cs programs.

This evaluation also found that all requirements for performance reporting were fully met. Programs completed reports, on some occasions in the absence of standard definitions and guidelines explaining how to use templates. The amount of information accumulated over time was enormous and the effort to compile and analyze the data will be enormous.

Lessons learned: Tools must be clearly defined and rationalized. It is useful to have an administrative policy, which would require that the purpose, objectives, and decisions underlying the tool implementation, are defined at the outset. Guidelines and standard operating procedures should be developed before the tool is used. Collaborative development and implementation of tools will promote the development of useable data.

These findings are also supported by the PMF (2006) report:

For the most part, data collection for the Federal Initiative is in the formative stages and is not automated; more work needs to be done to streamline reporting requirements and formalize responsibilities... Overall, performance reporting duties appear to be under-resourced… Some information is being collected on all of the indicators that have associated targets; however, the extent to which a baseline could or should be established for these indicators is unclear.” Further study is required to determine how much baseline information is required in general and how much is available.

Conclusions: Governance and Performance Measurement

The RCC has the potential to be an excellent model of horizontal initiative governance. Overall, the RCC is working collaboratively and each RC is making decisions that are consistent with its own area of responsibility and role. The RCC should be further supported to strengthen its role as the governance lead for the Federal Initiative. It would be useful to redefine RC roles in the context of both horizontal and vertical responsibilities.

As the RCC lead, the HIV/AIDS Division will need to focus efforts on establishing shared priorities and promoting better alignment with Federal Initiative requirements. RCs will need to work in collaboration with the HIV/AIDS Division to invest the necessary efforts required to advance Federal Initiative implementation.

There are challenges ahead. RCC planning processes will be called on to respond to changing policy expectations and funding reallocations. Horizontal management requires a clear understanding of organizational structures. A basic foundation for results measurements is a logic model that identifies the set of related activities and shows the chain of results connecting activities to final outcomes. The logic model is a graphic representation of the connections between program activities, immediate outputs, and anticipated outcomes. The model guides the development of a results-based performance measurement system that efficiently measures the progress of each component of this results chain. Developing performance measurement systems is a complex process. Implementing frameworks for horizontal initiatives takes longer than for vertical programs.

Federal Initiative partners should confirm that the necessary resources are available to undertake the performance measurement and reporting systems implementation. A review of the current logic model and an alignment of program indicators are priorities for the RCC. Engaging Federal Initiative working groups and programs to ensure performance measurement and reporting system implementation is a priority for success.

The governance mechanisms are assessed as implementation in progress. The RCC is an excellent structure, but a decisional structure without connections to tools and information is not effective. The RCC needs to be supported by a reporting system that effectively monitors Federal Initiative delivery. A performance measurement strategy is in place, but requires enhanced tools and an implementation strategy to facilitate data collection, analysis, and reporting.

C. Success

This section addresses the remaining two questions regarding progress towards achieving immediate outcomes and unintended results.

Q5: What progress has been made towards the achievement of immediate outcomes?

The data collected for this study provided evidence that Federal Initiative program activities and outputs are situated in the five Areas of Action of the logic model, and progress towards outcomes can be inferred. For example (from 2006-2007 DPR information):

Increased knowledge and awareness may be said to be reached through the following:

• development of national surveillance reports (SRAD);
• funding of 299 research grants and awards directly related to HIV/AIDS and 17 Canada Research Chairs in the area of HIV/AIDS (CIHR);
• support for community-based projects, such as “GUYZ”, which increases understanding and awareness of HIV, hepatitis C (HCV) and sexually transmitted infections (STIs) among young gay men (aged 19 to 29) and service providers in the St. John’s area (ACAP);
• development of data standards for sexually transmitted and bloodborne infections to improve national data quality and timeliness (CAID); and
• development of M-Track to achieve a better understanding of the risk behaviours involved in acquiring HIV co-infection with viral hepatitis and/or sexually transmitted infections. As a second-generation surveillance system, M-Track goes beyond disease reporting to try to understand changes in the epidemic and the behaviours that precede infection (SRAD).

Enhanced multi-sectoral engagement and alignment is reached through the following:

• support for advisory bodies and coordinating committees (HIV/AIDS Division).

Increased individual and organizational capacity through the following:

• discharge planning guidelines that provide support for inmates with ongoing care and treatment needs for infectious diseases, while under the community supervision portion of their sentence (CSC);
• HIV/AIDS and hepatitis C guidelines for nurses working on reserve (FNIHB);
• support for community-based AIDS organizations and projects. (For example, one project involved hiring a street youth to coordinate production of a food guide and recipe book for Montreal street youth living with HCV, which shares common modes of exposure with HIV. In addition to providing exemplary leadership, this individual encouraged a number of other street youth to participate in the project, some of whom were later able to find jobs.) (ACAP); and
• the Labs Quality Assurance Program supports provincial and territorial laboratories in ensuring the practice of uniform standards of HIV diagnosis in Canada (NHRL).

Increased coherence of the federal response through the following:

• effective coordination of the federal presence at the 2006 XVI International AIDS Conference in Toronto (IAD); and
• establishment of an interdepartmental ADM committee to address determinants of health related to HIV/AIDS across the federal government (HIV/AIDS Division).

While these activities may be beneficial, the tools have not been put in place to measure Federal Initiative progress to results. In addition, it is difficult to measure improvements against targets, which are not realistic. A“50% reduction in domestic incidence of HIV” requires the applied efforts of all governments and non-governmental stakeholders to be successful, and is not feasible or measurable as a target for a federal program. Federal Initiative targets should be revised to represent appropriate results for federal programs.

In conclusion, a focused exercise to align program outputs with outcomes and set realistic targets will create a fully operational logic model that can be used to demonstrate progress.

Q6: What were the unintended results from the implementation of the Federal Initiative?

The following are some unintended results that emerged during the evaluation study:

1. Vaccine research in 2004 was remarkable in raising expectations that the potential to contribute to HIV prevention was high. In 2006 the Canadian HIV Vaccine Initiative (CHVI) was established. Between 2006 and 2008, $3.7 million was transferred from the Federal Initiative budget to support the prevention work of the CHVI. Federal Initiative planned support for the CHVI over the five fiscal years from 2007/2008 to 2011/2012 will be $15 million.
2. After budget adjustments that took effect beginning in 2007-2008 (combining federal government-wide reductions and reallocations), the expected $84.4 million ongoing resources to fund Federal Initiative activities will not be reached. In 2008-2009 the available budget is approximately $72.6 million. This reduction had implications for engaging staff with the skill sets to work in performance measurement and evaluation and will place more onus on the RCC to identify joint priorities in future.
3. A CIHR-funded study found that people at risk of HIV become anxious the longer they test negative, believing their high-risk behaviour is safe. The results underscore the need for enhanced counselling for those who repeatedly test negative for the virus and continue to engage in high-risk behaviour.
4. The process of working on this evaluation created a dynamic environment among programs by increasing interactions and information sharing, and building capacity through joint ventures.
5. The learning process takes more time that expected. Horizontal management is relatively new to the federal government and there is a gap in external expertise to help guide the process.