Integrated Intelligence Evaluation Report

Background

1. In December 2016, the Public Service Commission of Canada (the agency)’s Internal Audit and Evaluation Directorate was asked to conduct a formative evaluation to provide advice on the potential future direction of the integrated intelligence function. As a result, this integrated intelligence evaluation was included in the agency’s Two-year Integrated Risk-Based Audit and Evaluation Plan 2017-2019.

2. Integrated intelligence is the process of synthesizing and interpreting the array of data, information and knowledge produced by an organization in order to:

• identify system risks
• inform decision-making and influence priority setting through insights, analysis and interpretation
• identify current and emerging trends — for example, in staffing

3. At the agency, integrated intelligence refers to a knowledge integration capacity that includes data governance, information collection methods, data analytics, data sharing and expert advice. Knowledge integration leads to the effective use of data, information and knowledge in support of strategic and operational decision-making.

4. As a component of its Enhanced Integration Project ^{Footnote 1} , the agency launched an integrated intelligence initiative in 2013 in order to:

• inform decision-making, program development and potential policy changes
• support outreach and oversight strategies through targeted information aimed at addressing specific needs
• influence priority setting (internally and externally) by sharing strategic insight, analysis and interpretation

5. Between January 2014 and mid-2016, the integrated intelligence initiative was relocated in several sectors of the agency. As a result, a number of activities took place to support this initiative, and several integrated intelligence products were delivered. This work was put on hold in July 2016, the same year that the Enhanced Integration Project resulted in the launch of the New Direction in Staffing.

6. There are concerns that information collected following the implementation of the New Direction in Staffing may not fully support the agency’s corporate reporting capabilities, and may not provide staffing support advisors with the information they need to fully assist departments and agencies in implementing the renewed staffing policy. The integrated intelligence evaluation also considered the agency’s Corporate Risk Profile 2017-18, which identified a risk that system-wide changes could impact the agency’s ability to meet its priorities in a timely manner.

7. In 2018, the agency created an Office of Data Management led by a Chief Data Officer. The office will play a key role in enabling the cultural shift that must occur for the agency to move towards a data-driven culture, and in gradually achieving incremental improvements in its organizational data management capability.

Objective

8. The objective of this formative evaluation of integrated intelligence is to assess the extent to which existing and proposed integrated intelligence work has:

• met the needs of the agency and its external stakeholders
• been designed and implemented across the agency

Scope

9. The evaluation covered the time period from December 2013 to April 2017. This period was selected to capture the work that was done from the start of the Enhanced Integration Project to the end of the 2017-18 fiscal year. The scope includes the following:

• internal and external stakeholders’ needs
• identification of the work done in the past
• availability and accessibility of existing intelligence
• recommendations for implementation

10. The evaluation was conducted in 2 phases:

• Phase I focused on assessing internal stakeholders’ needs related to integrated intelligence (completed in March 2018)
• Phase II focused on identifying and contributing to a better understanding of external clients’ needs related to integrated intelligence (completed in April 2018)

Scope limitation

11. For the purpose of this evaluation, integrated intelligence is considered as a function, as it is not a program in the formal sense. As a result, the performance and effectiveness of integrated intelligence to date was not examined. Although the scope of this evaluation includes the examination of the modifications and alternatives for the design of an integrated intelligence function, providing a blueprint for this function is beyond the mandate of this project.

Methodology

12. The evaluation used a mix of quantitative and qualitative methods to incorporate multiple sources of evidence. The following methods were used (see details in Annex – Methodology and sources):

• document review and analysis
• literature review
• interviews with internal and external stakeholders
• survey of the agency’s stakeholders
• survey of heads of human resources (HR) and corporate HR advisors across federal public service departments and agencies
• analysis of administrative data

Findings

13. This section presents the major study findings, organized by evaluation issue. For each issue, the results of the review of key documents, the literature review (where relevant), and consultations (interviews and surveys) with internal and external stakeholders, are presented.

Need for an integrated intelligence function

14. The evaluation was not able to find an integrated intelligence function charter at the organizational level that establishes a common definition, objectives and a monitoring or measurement plan for the development and implementation of the function.

15. The need for integrated intelligence at the agency was first identified in 2013 and has increased with the introduction of the New Direction in Staffing in 2016, which led to a shift in federal public service staffing:

• from a prescriptive model that provided specific policy guidance and advice
• to agile approaches that give departments greater ability to customize staffing systems, based on their realities and needs

This change led the agency to discontinue various oversight tools, such as the Staffing Management Accountability Framework, the Departmental Staffing Accountability Report and departmental specific staffing audits. The loss of these tools may have reduced the agency’s capacity to monitor the health of the staffing system and to take appropriate action.

16. No evidence was found to show that a systematic process was in place within the agency to address the needs, gaps and risks that were identified in previous work (including the Data Services and Analysis Directorate’s consultations in 2015, a consultant’s work in 2016 and a proposal for the development of an integrated intelligence function) in order to improve integration of intelligence.

17. As the agency’s needs in the area of data and information analytics continue to grow in number and complexity, a robust information infrastructure would be an important asset to ensure the agency has the strategic intelligence to identify environmental and system trends and risks, to support informed evidence-based decision-making.

18. Most of the internal stakeholders consulted for this evaluation identified a need for an integrated intelligence function. They indicated that the strategic use of intelligence would support the agency’s accountability to Parliament by providing the President with accurate, timely and reliable information and data about staffing strategies across the federal public service. A well-functioning integrated intelligence function would also provide the agency’s senior management and deputy heads across the public service with data and information driven insights that could contribute to changes in policies, strategies, programs and legislation related to the federal staffing system.

19. Integrated intelligence could also support the agency’s objectives related to implementing the New Direction in Staffing, by enhancing staffing monitoring and by identifying potential actions based on the integrated analysis of emerging trends and risks across the staffing system.

20. Based on the external survey sent to HR representatives in departments and agencies, 55% of clients indicated that the information the agency provided on implementing the New Direction in Staffing was useful for hiring managers. Those who did not agree pointed out a lack of in-depth and concrete information, including best practices and scenarios that they could learn from

Intelligence awareness and sharing

21. Currently, various intelligence holdings exist within the agency, including information on the health of the staffing system, investigations and non-partisanship reports, statistical staffing data, priority administration, policy analysis, and internal and external communications. While knowledge sharing at the strategic level (for example, at Executive Management Committee) has been formalized, it happens on an ad-hoc basis at the operational level. This primary data was corroborated by a review of the implementation of the New Direction in Staffing, conducted by the Policy and Communications Sector in 2018. The review recommended, among other things, improving cross-functional communications and engagement within the agency.

22. Given the ad-hoc nature of information and knowledge sharing at the operational level, the existence of all available intelligence products is unknown. The internal stakeholder survey indicated that most respondents (90%) were at least somewhat aware of the agency intelligence holdings, and 77% had various degrees of access to the holdings. Awareness and access related primarily to intelligence holdings within an employee’s specific sector.

23. In 2013, the Enhanced Integration Project team observed that there was no systematic sharing of intelligence within the agency. More than 4 years later, this evaluation found that there has been limited progress, as only 40% of internal survey respondents indicated that the current sharing of intelligence was developed or well developed.

24. Internal interview respondents expressed a clear understanding of how integrated intelligence may contribute to fulfilling the agency’s mandate. Most of them raised the lack of a dynamic knowledge-sharing culture as a barrier to achieving that objective. The evaluation found that there is a need to develop a culture of information sharing embedded in the agency’s governance function to provide employees at all levels of the agency with a clear understanding of how they can share intelligence.

25. In terms of the organizational culture, some interviewees indicated that there was a tendency for employees to only infrequently seek information from outside their sector. Interviewees and survey respondents also indicated that the focus on day-to-day activities and, in some cases, lack of technological skill, may prevent employees from fully participating in intelligence sharing and integration.

26. In order to build a dynamic information-sharing culture, there is a need to promote intelligence sharing between sectors and directorates. Inter-sector and regional exchanges of information and knowledge would strengthen the information presented for decision-making, decrease duplication of effort, and help develop a comprehensive view of the staffing system across the federal public service.

27. While most external survey respondents were aware of the existence of the agency’s information and data products, about 50% indicated they were not aware of the specific location of these sources of information. The external survey found also that 29% of the respondents in the HR category had difficulty finding information produced by the agency. The survey also showed that during the transition of the agency’s website to Canada.ca, information became more difficult to find.

28. Opportunities for improvement in how information is organized were also identified by internal stakeholders. Most interviewees and survey respondents mentioned that the lack of a standardized approach across information and data holdings was a major roadblock to information integrity and knowledge integration. The quality of information provided by the agency to clients was also raised as an issue in survey responses, especially by internal clients. This leads to a risk of inaccurate and misinterpreted information being produced and released within and outside the agency. This also affects timely analysis and the identification of trends and risks in the staffing system, and in implementing the New Direction in Staffing.

Integrated intelligence design and governance

29. Before the launch of the Enhanced Integration Project in 2013, there were tools and forums to strengthen integrated intelligence within the agency. There was a Research and Studies Network (co-chaired jointly by the Policy Branch, and Audit and Data Services Branch officials) that aimed to facilitate information-sharing, coordination, and possible collaboration on research projects to support developing a coherent and integrated approach to addressing organizational priorities.

30. In addition, the Ingenium software application was introduced to give the agency’s employees the ability to search and analyze various data holdings. At the time this evaluation was conducted, employees had stopped using this application. The review of the literature stresses the importance of business intelligence information technology (IT) systems in enabling information and data sharing, as well as data quality and integration capabilities. Since the decommissioning of Ingenium, the agency has reviewed various business intelligence and enterprise data warehousing solutions. A number of projects are currently being developed that aim to address data integrity and access concerns and related IT components, through the implementation of the agency’s Data Management Strategy.

31. Two formal initiatives existed between December 2013 and December 2017.

• The first took place between 2014 and 2015 and was carried out by the Horizontal Integration Directorate. Its objective was to ensure that intelligence was shared and integrated at the operational level by bringing together policy and data research to support a culture of evidenced-based decision-making. Its main governance bodies were the Integration Committee, and the Forum for Integration and Exchange, which replaced the Research and Studies Network, but never came into force.
• The second initiative was led by the Data Services and Analysis Directorate after 2015. The goal was to develop and provide data products for both internal use (for example, organizational profiles, in collaboration with the Staffing Support and Priority Directorate) and external clients (for example, staffing dashboards). The Data Services and Analysis Directorate’s value proposition, in the context of the New Direction in Staffing, was to offer internal and external clients data self-service tools to make staffing system monitoring and reporting more efficient.

32. The evaluation team analyzed the integrated intelligence initiatives that have been developed within the agency through the lens of knowledge management practices identified in the literature review. These knowledge management practices are categorized into 2 key types of activities:

• primary activities, which consist of sharing, integrating and actioning intelligence
• secondary activities, which consist of acquiring, transforming and disseminating intelligence

33. Based on this analysis, integrated intelligence initiatives identified within the agency focused more on secondary activities than on primary activities. This further supports the need for the agency to focus on building a knowledge and information sharing culture that would support the full implementation of a robust integrated intelligence function.

34. Together, the agency’s Open Government Advisory Group and its efforts to build an integrated intelligence function have potential synergies, in the medium term, to leverage the agency’s data as well as information sharing and dissemination. In April 2018, the agency decided to strengthen the capacity of the Office of Data Management and Open Government Secretariat, which is a positive step in building a knowledge integration capacity throughout the agency.

35. According to the literature, knowledge management initiatives are often driven by senior management with the help of a centralized office, led by a Chief Knowledge Officer or a Chief Data Officer. According to a recent Gartner report, “Chief Data Officers are the head custodians and chief stewards of an organization’s information and assets. Organizations have a growing stake in aggregating information and using it to make better decisions. Chief Data Officers are tasked with using information to automate business processes, understand customers, and develop better business relationships with partners.” However, the literature finds that co-locating this office in an IT setting is usually counterproductive, as the IT environment focuses on certain aspects of data management rather than knowledge creation.

Integrated intelligence governance

36. The evaluation team learned through interviews and literature reviews that a robust integrated intelligence function starts with organization-wide coordination as well as appropriate information collection, sharing, integration and reporting processes. The agency collects and stores information from multiple internal and external sources. It must build a knowledge integration capacity to help monitor and safeguard the integrity of the staffing system.

37. The survey responses, interviews with officials from other departments and agencies, and the review of documents revealed a best practice: a multi-sector planning and coordination committee that can identify high-level strategic intelligence needs, to support intelligence integration across the agency and build a knowledge-sharing culture.

38. The evaluation found that all of the agency’s sectors are involved in intelligence gathering, production, and reporting. For example:

• the Policy and Communications Sector collects information on government-wide staffing and overall trends, issues and risks in implementing the New Direction in Staffing
• the Oversight and Investigations Sector focuses on providing analytics on the integrity and effectiveness of the federal public service staffing system, including the protection of merit and non-partisanship
• the Corporate Affairs Sector provides the full range of information management, financial and HR information
• the Services and Business Development Sector is responsible for the Public Service Resourcing System and provides expert assessment products and services to hiring managers, while also administering central recruitment programs, regional offices and second language evaluations

39. While each sector focuses on collecting and understanding information related to its mandate, there is no evidence that a governance structure has been fully developed to support integrated intelligence at the agency.

40. Most survey respondents agreed that the Integration Committee could be a significant intelligence-sharing forum. However, interviews with sector management committees, as well as the survey results, noted that the committee is not being fully used to promote integrated intelligence discussions, which is one of the factors affecting the progress of the agency’s overall integrated intelligence function. It should be noted that, based on its terms of reference, the Integration Committee’s mandate is to facilitate consultations and information sharing between the operational and strategic levels.

41. The evaluation found that until now, there has not been a proper governance structure in place at the agency to support integrated intelligence. A governance structure would support the definition of roles and responsibilities related to the coordination and participation of all sectors in integrating quantitative and qualitative information from across the agency. The agency needs all sectors and regions to participate in integrated intelligence to ensure that knowledge sharing, integration and actions are optimized to support evidence-based decisions.

42. There are a number of governance structures that could strengthen integrated intelligence at the agency. These include:

• a centralized approach housed in one business unit
• a decentralized approach with intelligence centres located in each sector
• a hybrid approach with a coordination centre that relies on ambassadors or satellites embedded in each sector

43. The literature review helped the evaluation team identify some of the advantages and disadvantages of the different governance models for integrated business intelligence. They are summarized in Table 1 below.

Table 1 – Centralization versus decentralization of governance models

44. As illustrated below, the hybrid approach is the preferred structure (37%) for the integrated intelligence function, followed by a centralized approach (30%). It is interesting to note that only 9% favoured a decentralized approach.

45. Interviewees who were in favour of a hybrid approach to integrated intelligence suggested that a coordination hub could be housed in a sector, such as the Policy and Communications Sector, with strong intra and inter organizational linkages to ensure the participation of all sectors, regions and the agency’s clients, to sustain the best information flows. This hybrid approach would:

• reduce inefficiencies due to information being scattered or duplicated across various sectors
• provide enough flexibility to allow for a natural flow of information to and from all corners of the agency, as well as to external clients
• establish an ambassador within each sector to help promote the culture change needed to support the flow of information and data across the agency

46. Many respondents also stressed the role that governance mechanisms could have in supporting integrated intelligence efforts to address current information sharing practices and to increase the flow of data and information across the agency.

47. During the planning phase of this evaluation, a number of agency officials indicated that there might be value in establishing a pilot project to test how an integrated intelligence function would work. They noted that any decision on the location of a pilot project should consider the project scope, its design and objectives. There was also a consensus that the pilot project needs to be housed within a horizontal unit, as described previously.

48. Some comments indicated that the suggested pilot project should be embedded in a sector in order to validate proof of concept before launching an organization-wide integrated intelligence project.

Good practices

49. To identify good integrated intelligence-related practices, the evaluation team consulted with other federal public service departments and agencies. The consultations revealed ongoing work across the public service to identify how knowledge, information and data can be used to:

• improve client service efficiencies
• assess and address risks
• improve analytical capacities and competencies to support strategic and operational decision-making processes

50. The evaluation found approaches that could be considered for implementation at the agency. The evaluation team will share this information with the office of primary interest for their consideration in developing an integrated intelligence function within the agency. A high-level overview of each model is presented below.

51. At the Public Health Agency of Canada, 4 groups were identified as data intensive. Representatives from each of these groups were brought together in a “4 corners” committee in order to submit more coordinated and strategic data requests, integrate data management, and align data intensive areas with requirements to meet government deliverology expectations.

52. The Department of National Defence has set up an Integrated Strategic Analysis team within the Chief Review Services, Evaluation Directorate, to facilitate assessments that span the 3 branches of the Canadian Forces. This team expects to expand the methodology to include internal audit in the near future.

53. The evaluation team learned through interviews with internal stakeholders that past iterations of integrated intelligence at the agency had considered the Canada Revenue Agency model. This model was based on automating monitoring in databases in order to send alerts when a risk is detected. This approach is similar to the model that was presented to the evaluation team by the Office of the Superintendent of Financial Institutions.

54. Other models in use across the federal public service were also identified by the evaluation team. The Canadian Transportation Agency uses an automated risk detection system to assist in compliance assurance. The Canada Border Services Agency published an evaluation report of their intelligence program in 2014. Recognizing the differences in the nature of intelligence at the Public Service Commission and Canada Border Services Agency, it is interesting to note that the report identified a need to establish a single centre of intelligence.

55. Transport Canada’s Strategic Policy and Innovation Directorate collaborates with internal partners and Policy Horizons Canada to conduct environmental scans (e-scans) of global and domestic trends that could have an impact on the transportation system. Environmental scanning is the process of gathering, analyzing and dispensing information for strategic purposes that entails monitoring changes in the internal and external environments that may emerge in the short- (less than 1 year), medium- (1-5 years) or long-term (5-20 years) by assessing political, economic, social, technological and environmental factors. The methodology draws on a combination of hindsight, insight and foresight. The e-scan process results in a comprehensive report that is used extensively throughout Transport Canada for different purposes, including business planning, risk assessments and the development of risk mitigation strategies. It is a key piece in departmental risk assessment and priority setting. It is also used in independent reviews in the area of transportation.

56. Health Canada and the Public Health Agency of Canada have recently put in place a Stakeholder Intelligence Service as a one-stop-shop. It provides in-depth understanding of stakeholders, resulting from quality analysis of stakeholder information, with the goal of supporting decision-making and engaging more effectively with stakeholders. Examples of studies include:

• short-term projects that meet a specific need for timely information relevant to a current issue
• analysis that is carried out from the whole departmental or health portfolio perspective
• analysis around basic departmental or publicly available information

57. Interviewees in other departments provided insight on elements which, depending on the operational context, could increase the usefulness of integrated intelligence. These elements include:

• working with an established risk framework
• integrating qualitative data into structured data
• putting in place data governance that rests on well-developed and maintained relationships with clients and partners

58. The integrated intelligence models used by other departments and agencies were grouped and summarized as shown in Table 2 below.

Table 2 – Federal public service integrated intelligence comparative analysis

Considerations

59. Any integrated intelligence function that is proposed for the agency should contribute to desired outcomes, including:

• helping the agency and its clients access reliable integrated data and information, so that they can make informed and timely decisions about the staffing system
• increasing understanding of staffing challenges and potential risk

60. The outputs of the integrated intelligence function may also be used to feed into strategic documents (for example, reports to Parliament), support consultations or engagement activities, inform stakeholders about the agency’s programs or policies, and advise senior management in preparing for meetings.

61. Over the course of the past 2 years, there have been significant changes in the agency’s operating context that have made integrated intelligence more relevant and timely. First, the New Direction in Staffing is being implemented and requires different monitoring approaches to ensure its effectiveness. Second, the agency is integrating and using data to support decision-making processes through the creation of a Chief Data Officer and the creation of the Data Management and Open Government Secretariat. In addition, in order to support the agency’s strategic decisions, it is time for the agency to leverage the information and data holdings of its regional offices, given their dealings with other government departments and agencies, public servants, and the public in general.

Success factors for the implementation of integrated intelligence

62. Success factors for implementing a knowledge integration capacity are based on removing organizational, cultural and workflow barriers to horizontal sharing and use of information. Removing these barriers would enhance collective knowledge and strategic and operational decision-making.

63. Making integrated intelligence a key element of the governance framework would enable the agency to support its mandate. The agency also needs:

• to properly identify the distinct and overlapping intelligence needs between sectors
• to ensure that sectors share their information holdings
• a more robust capacity to systematically undertake meta-analysis and share the results in a meaningful and timely way

Recommendations

Conclusion

64. Given the agency’s mandate and the important role that it plays in federal public service staffing, there is a need to ensure that the information and data collected by the agency are understood and used to support evidence-based strategic and operational decision-making. In light of the New Direction in Staffing and its impact on the agency’s delivery model, which is now based on monitoring the health of the staffing system, there is an increased need for more integrated intelligence to improve knowledge sharing, to gain early awareness of environmental changes, as well as to manage organizational risks. Finally, the results of this formative evaluation indicate that the intelligence and knowledge integration capacity within the agency should be supported by a strong governance framework and based on a culture of information and knowledge sharing.

Management response and action plan

Annex – Methodology and sources