IT Client Services - Applications Development and Maintenance Process
Final Audit Report
June 20, 2012
Table of contents
- Executive summary
- A. Background
- B. Findings, recommendations and management responses
- 1.Governance and organization for service delivery
- 2. IT Business solutions and needs analysis
- 3. IT Business solutions development and maintenance process
- C. Conclusion
- Appendix A - Lines of enquiry and criteria
- Appendix B - Scorecard
- Appendix C - Application Delivery Framework
The objective of the audit was to determine if Health Canada has a process in place to effectively and efficiently build and maintain information technology (IT) applications. The audit was conducted in accordance with the Treasury Board Policy on Internal Audit and the International Standards for the Professional Practices of Internal Auditing. Sufficient and appropriate procedures were performed and evidence gathered to support the audit conclusion.
The information management/information technology (IM/IT) budget for application development at headquarters for the fiscal year 2011-2012 was $15.4 million. For the same time period, Regional IT application development and maintenance was reported at approximately $0.6 million, however most application development is contracted outside the Department. The branches have allocated $49.4 million for application development projects approved in the Health Canada Investment Plan for 2011-2012.
Overall, while the Department has practices to develop applications and to maintain them. This activity will benefit from strengthening the governance, risk management and internal controls.
As a result of significant changes across the Government of Canada as well as within Health Canada, the Information Management Services Directorate recognizes the need to re-evaluate its IT governance and service delivery model for application development and maintenance. Given that the departmental Investment Planning Process has a documented and approved governance model combined with approved and documented project management practices, the Department would be well served if applications, either under development or planned, were delivered as part of the Investment Planning Process. Using this process to support application development will provide: oversight mechanisms to ensure that controls surrounding the development and maintenance of IT applications accurately reflect departmental objectives; well defined roles and responsibilities; and gating mechanisms that can be used to mitigate business risks and drive better discipline around project management.
The portfolio of IT applications for Health Canada could be based on a few factors such as the functional business environment; a determination of current and future demand; and an analysis of the development and maintenance cost and risk. In the past this analysis has been done in a stove-piped manner - branch by branch. However, there are functions that cut across several branches and the Department would benefit from documenting a set of information which, collectively, can be used to identify business needs, investment priorities and to define the Department's enterprise IT architecture. The models should provide a common language for the Department; support the identification of duplicate, re-usable and sharable applications; provide a basis for objective review of IT investments; and enable more cost-effective and timely delivery of IT services through a repository of standards, principles and templates as expected in a system development lifecycle approach.
Establishing the Department's system development lifecycle methodology with updated controls and guidance combined with leveraging regional capacity to complement headquarters will serve to strengthen the enterprise approach while ensuring applications are efficiently and effectively developed and maintained.
Service level maintenance agreements for existing and new applications will provide branches with business continuity reassurance as well as having the Information Management Services Directorate develop performance metrics for maintenance. While the Directorate has been actively reducing the number of applications in the Department, they will need to expand this initiative to form a longer term IT plan outlining how the Department will manage new application priorities in conjunction with maintaining legacy systems in order to deliver the best IT services.
Management has agreed with the five recommendations and provided a detailed action plan that, once implemented, will strengthen the application development and maintenance process at Health Canada.
As a science-based department with a largely professional workforce, Health Canada needs applications, tools and infrastructure to facilitate the delivery of its regulatory and policy mandate. Effective application development and maintenance processes are critical to the success of information technology (IT) projects. Application development and maintenance methodology should provide the processes to ensure highly successful, on-time, low cost business solutions. Services often include architecture consulting, capacity planning, network planning, development and testing.
(see Appendix C)
Application Development Activities
- Define the solution architecture
- Define requirements (including use of prototypes)
- Design the framework of the solution
- Build the solution
- Validate the solution against requirements
- Roll out the solution
- Continuous support for the solution
The budget for information management/information technology (IM/IT) activity in headquarters for the fiscal year 2011-2012 was $79.6 million; of that, $15.4 million was allocated to applications development and maintenance. The branches have allocated $49.4 million for application development projects approved in the Investment Plan and Regional IT activity is reported at approximately $0.6 million.
IM/IT activities within Health Canada underwent a major reorganization from 2005-2006 to 2006-2007 to adopt an enterprise approach for IT management. This activity centralized the responsibility for Health Canada's IT hardware, software and services under the Assistant Deputy Minister, Corporate Services Branch in order to generate savings. This initiative involved the consolidation and realignment of Health Canada's resources and infrastructure, the development of departmental standards, and the repositioning of the Department to leverage and align with the Government of Canada's common services initiatives. Five years on, Health Canada is still in the process of fully rationalizing the complexity that arose as a result of the branches and regions independently developing applications to meet program requirements. Multiple generations of software are in use, such as the Oracle database package, which adds to the complexity of operations. The Application Reduction Initiative within Health Canada has reviewed approximately 2,000 applications of which almost 800 have been identified for archive and disposition.
1. Audit objective
The objective of the audit is to determine if Health Canada has the processes in place to effectively and efficiently build and maintain IT applications.
2. Scope and approach
The engagement was carried out within Health Canada's National Capital Region and in the regions across the country. The audit examination was limited to IT processes in existence at the time of audit (fiscal year 2011-2012) and historical information on projects in process. In addition to work conducted in the National Capital Region, the approach involved site visits to the Atlantic, Ontario, Northern, Manitoba, Saskatchewan, Alberta and British Columbia regions.
Control Objectives for Information and related Technology (COBIT) was used as a source of audit criteria outlined in Appendix A. COBIT is a widely-used framework promulgated by the IT Governance Institute, which defines a baseline of computer application control objectives with recommended approaches as to how such objectives may be evaluated.
The focus for the Audit of IT Client Services - Application Development and Maintenance Process included:
- Governance and organization for service delivery - sufficient oversight mechanisms exist to ensure that controls surrounding the development and maintenance of IT applications accurately reflect departmental objectives; roles and responsibilities reflect an appropriate level of segregation of duties; and monitoring mechanisms are established to mitigate business risks.
- Application needs analysis - Health Canada develops a portfolio of IT business solutions based on an understanding of the client business environment, a determination of current and future demand and an analysis of the cost and risk of solution development/maintenance.
- Application development and maintenance - The development and maintenance process has been designed to best meet the needs of business users and corporate priorities within the selected cost envelope.
The examination phase involved interviews with the staff of the Information Management Services Directorate (IMSD), regional Information Management Services (IMS) staff and with key representatives of IMSD clients at the national and regional levels. The engagement also included an examination of selected process and project documentation, operating procedures and tools.
3. Statement of assurance
In the professional judgment of the Chief Audit Executive, sufficient and appropriate procedures were performed and evidence gathered to support the accuracy of the audit conclusion. The audit findings and conclusion are based on a comparison of the conditions that existed as of the date of the audit, against established criteria that were agreed upon with management. Further, the evidence was gathered in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.
B. Findings, recommendations and management responses
1. Governance and organization for service delivery
1.1 Strategic direction
Audit criterion: The Department has defined an enterprise architectural strategy for application development aligned with corporate priorities.
Within an enterprise IT management strategy, determining which IT applications to develop should be guided by departmental priorities and aligned with its architecture principles. Architecture principles are the general rules and guidelines established by the Chief Information Officer for the use and deployment of IT resources and form the basis for making future IT decisions.
Health Canada recently conveyed its strategic direction for application development through the Investment Plan. For fiscal year 2011-2012, Health Canada's Investment Plan consists of 18 IT projects for programs and 10 IT projects for corporate services. The list was pared down from 2010-2011 and will continue to be refreshed as the Department continues to find the right balance amongst competing business needs and limited resources.
Across the federal governments of the United States and Australia, departments and agencies are opting to develop IT solutions that are based on common business processes. This has come about largely from pressure to improve client satisfaction and organizational productivity. As such, strategic business modeling is becoming steadily more ingrained. At one time, for example, models were produced only for specific automation and process improvement purposes. Now, "whole-of-enterprise" strategic business modeling is becoming more common. It has the advantage of putting a multitude of different processes into context showing where they are inter-dependent; so that organizations can reduce the duplication of applications, re-use applications and ensure that new applications are shared. Mapping common business process across the branches would be beneficial in order to confirm architecture principles and to target investment in a strategic set of applications.
It is recommended that the Assistant Deputy Minister, Corporate Services Branch, in consultation with client branches, define an enterprise architectural strategy for application development.
Management agrees with the recommendation.
The development of business reference models will establish a long term vision and will describe Health Canada's activities around common business functions (such as regulatory).
These models will promote departmental collaboration by establishing baseline information. This will provide the opportunity for the Department to collectively determine priority investments.
1.2 Management oversight
Audit criterion: Appropriate oversight has been established to ensure IT applications are developed and maintained in accordance with corporate priorities.
Application development and subsequent maintenance requires an established governance structure within which decisions occur to provide for an enterprise architecture approach including guidance for the selection, management and completion of projects.
Project selection was a bottom up process that began with the various Branch Executive Committees approving a short list of major IT projects. The list was submitted for review to the Director General - Investment Planning Committee - a forum for consultation on all investment planning issues, processes, projects and activities. It ensures consistency with strategic direction and alignment with Treasury Board policy. Committee members are also responsible for communicating investment planning decisions within branches. The Committee is comprised of representatives from all branches. Recommendations from this committee are subsequently reviewed by the Assistant Deputy Minister - Investment Planning Committee. This Committee reviews the Investment Plan and projects for alignment with business directions. It provides oversight and recommendations to the Executive Committee. It ensures projects are consistent with strategic directions, Treasury Board and Health Canada policies. The Committee is composed of select branch assistant deputy ministers and is co-chaired by the Chief Financial Officer and the Assistant Deputy Minister, Corporate Services Branch. Final review and approval is completed at the Executive Committee - the most senior level decision making body.
Within the Information Management Services Directorate the Operations and Oversight Committee acts as the monitoring point for all IT activities in the Department including projects in the Investment Plan. At the time of the audit, the IT Project Delivery Office was responsible to provide secretariat duties for these three oversight committees.
Prior to the Investment Planning Process, Health Canada established its IM/IT priorities through the annual operational planning process leading to a portfolio of projects, known as the IM/IT Plan of Record. Although branch and regional staff interviewed clearly understood the benefit of making strategic decisions on IT investments via the Investment Planning Process, they are unsure as to the status of projects previously on the Plan of Record that are not reflected in the Investment Plan as a result of these projects falling below the investment planning thresholds. According to Treasury Board policy, projects with costs exceeding $1 million are to be included in the Investment Plan and require a Project Complexity and Risk Assessment to determine the level of project management oversight.
Currently, the existing governance for application development does not consider these important IT projects that fall below the investment planning threshold. In addition, several regional IT application development initiatives are not covered by the current governance structure. Since the Department has developed a risk-based project management process which includes varying levels of oversight by investment planning governance bodies commensurate with the financial value, the complexity and the risk level of each project, it would be beneficial to have all IT projects subject to the rigour of the Investment Planning Process.
In addition, it would be important for the Informational Management Services Directorate to govern the applications deployed via sustain strategies, enhancement strategies or application retirement strategies. Currently, many of these activities are managed as a part of the change management process within the IT directorate.
It is recommended that the Assistant Deputy Minister, Corporate Services Branch establish a governance framework for all application development and maintenance.
Management agrees with this recommendation.
Management has already begun to use the Investment Planning and Reporting Framework to support application development.
The Investment Planning and Reporting Framework was recently approved by the Health Canada - Executive Committee. This governance model identifies common business requirements related to projects.
The Information Management Services Directorate will also strengthen its internal governance model for application and development and maintenance project management.
The Information Management Services Directorate will define governance for both the Investment Plan and the Sustain (in-service) application maintenance activities.
1.3 Roles and responsibilities
Audit criterion: Responsibilities and accountabilities related to IT application development and maintenance have been clearly defined, documented and understood.
Providing and making use of IT services is a partnership between the Information Management Services Directorate and the consumer of the services, namely the branches. To make best use of limited resources, all corporate and regional parties must understand their role, and an established structure must exist under which development and maintenance expertise can be leveraged across the Department.
The Information Management Services Directorate has been subject to and is still undergoing significant evolution. According to roles and responsibilities descriptions obtained at the outset of the audit, the Enterprise Architecture Program was created to develop standards and to provide analysis and advice to align IT investments with business directions based on a long-term departmental strategy. This group has now been reorganized to integrate the IT architects into other activities leaving the Chief Enterprise Architect reporting to the Chief Information Officer. For some time, this role was vacant; a new Director of Enterprise Architecture has recently been staffed.
Part of the Directorate's approach to managing business requirements has been the involvement of the Client Engagement and Governance Group. The Group is responsible for providing strategic planning advice to Health Canada's branches on IM/IT services and investments, including new application development and departmental infrastructure initiatives.
As previously mentioned, the Project Delivery Office responsibilities include secretariat duties for three oversight committees. The Director in charge also has responsibilities for enterprise architecture and client portfolio services. These responsibilities are undergoing further clarification as they have been expanded to include other business functions.
In the IT industry, strategic planning normally resides in the conceptual world of the architect, not the client outreach function. Client outreach identifies the demand and architecture sets the forward direction. IMSD needs to rebuild its capability to foster an architectural vision for the Department, and monitor IM/IT activity against that vision.
Regional Information Management Services (IMS) is the regional supplier of IT services and during the audit was reporting to the Assistant Deputy Minister, Regions and Programs Branch. Given that the headquarters team reports to the Assistant Deputy Minister, Corporate Services Branch, roles and responsibilities have not always been clear between headquarters and the regions. The issue of what is "local" versus what is "national" should be redefined especially in light of the recent creation of the new Shared Services Canada Agency. While roles are defined within IMSD, the headquarters team needs to establish better communication with regional IMS staff and their client branches.
Although each region has IT support, they each operate differently regarding IT staffing, retention, funded positions and the degree to which IT staff communicates with regional programs. For example, one region had a working agreement that allowed IT resources to report to IMS but were funded by the First Nations Inuit Health Branch under a three-year Memorandum of Understanding, another region has several very experienced but underutilized IT staff and a third region exhibited the best example of employing both senior and junior IT staff on headquarter projects through local agreements.
The Assistant Deputy Minister, Corporate Services Branch and the Chief Information Officer visited several regions over the past two years to discuss regional IMS participation in application development. Interest was expressed by every senior regional manager regarding an approach to application development that would better utilize regional capability. The regions indicated that they could provide better value through a joint headquarters/regional approach given their inherent understanding of regional business processes combined with their IT expertise. Leveraging regional expertise would address regional dissimilarities as well as complement IMSD capability in filling departmental business requirements.
Branches are the business owners of the applications and must work as a partner with IMSD and regional IMS. Once an application is approved for development, the business owners (branches) have a responsibility to develop the business requirements and to work with IMSD to meet departmental architectural standards. Although IMSD has the Enterprise IT role, branches have, in the past, taken on a noticeable autonomy and authority. Given IMSD's reduced capacity, branches have proceeded with application development by procuring the necessary IT skills. These types of actions make it increasingly difficult to continue to put in place an enterprise approach. Increasing branch independence means that IMSD and regional IMS staff cannot provide a challenge function to ensure that departmental application development standards are being met.
Effective communication and clarification of the roles and responsibilities between IT and programs regarding required services is best enabled by a formal agreement such as a Service Level Agreement. Signed agreements typically reflect: all critical IT services required linked to program mandates; IT capabilities; appropriate funding levels; measurement and reporting of project performance against key project performance scope; schedule; quality; cost; and risk criteria. The monitoring and timely reporting to clients on the accomplishment of service levels better enables the alignment between IT services and the related program mandates. IMSD has service level agreements in place with its external clients but does not have separate agreements with the branches. Unlike headquarters, some regions have service level agreements in place.
Departments and agencies use a Service Level Agreement both for internally resourced services and for third party partnerships. The main reason departments and agencies enter into service level agreements is to improve the effectiveness and efficiency of service delivery in addition to the fact that negotiated agreements provide clear and documented measures of performance. Since IT applications being purposed or developed can rely on the Investment Planning Process for reporting, it would only be necessary for IMSD to consider service level agreements to capture the maintenance needs of the branches.
It is recommended that the Assistant Deputy Minister, Corporate Services Branch develop service level agreements with branches for the maintenance and support of applications.
Management agrees with this recommendation.
Based on existing standards, the Information Management Services Directorate, in consultation with clients in Health Canada/Public Health Agency of Canada, will review all applications and ensure that they are categorized by risk.
The Information Management Services Directorate will review existing service standards and update them to address any gaps. The appropriate service level standard will then be identified and confirmed with application owners.
1.4 Monitoring and management reporting
Audit criterion: Management has established regular monitoring specific to the development and maintenance of IT applications to report on the status of: planned objectives; performance targets; solutions delivery; and client satisfaction.
The IT Project Delivery Office is responsible for reporting and monitoring of projects via two reporting tools: the Project Information Reporting Tool and the Application Development Request form. The monitoring activities involve the collection of bi-weekly status reports from IMSD project managers. Project status information is compiled into a project dashboard that is used as the main feed for Investment Plan portfolio reports. Beyond the investment planning reporting requirements which are recorded separately, the Operations and Oversight Committee also identifies other projects to track. Maintenance activities are captured in another part of the Project Information Reporting Tool but are not rolled up into the dashboard as the project reporting criteria specifies.
Monitoring and management reporting for application development and maintenance is tied to the project management discipline imposed by the Project Delivery Office and its reporting tool. (see recommendation 2) The audit found that the Investment Plan project management discipline involves appropriate management reporting and monitoring.
2. IT business solutions and needs analysis
2.1 Business requirements assessment
Audit criterion: Business requirements should be identified prioritized and maintained covering the full scope of all business needs to be addressed.
The development of a new application requires analysis before acquisition or creation to ensure that business requirements are satisfied in an effective and efficient manner. This process covers the definition of the needs, consideration of alternative sources, review of technological and economic feasibility, execution of a risk analysis and cost-benefit analysis, and conclusion of a final decision to 'make', 'partner' or 'buy'. All these steps enable organizations to maximize the return on investment to acquire and implement solutions while supporting the business.
IMSD has the needed elements to capture, manage and assess business requirements but does not have a "process" as defined above; IMSD has a basic framework for application development and a set of standard templates - which, taken together, form a practice. A spreadsheet (also called the milestone grid) outlines deliverables, accountability and expected signoff covering an entire system development. However, no overall process documentation was found. In addition, two separate sets of practices exist - one for new system requirements and one for ongoing maintenance.
Branches that partner with IMSD to have applications developed are provided with templates designed by the Business Analysis Unit. While the Branch is accountable for the contents, IMSD often documents the requirements with client branches. In some cases, branches have their own business analysts and/or external contractors who develop the requirements. Regardless, the Business Analyst is expected to use standard toolsets and procedures and IMSD has a role to monitor adherence to the standards before going into development phase. The Business Intake Team validates the application development requests and evaluates things such as technology and the funding available to determine whether or not it will be approved.
The audit sampled 13 application development projects to test the use of the lifecycle approach. Audit results revealed that the process is not always clear as projects vary in the use of the standard templates however, lessons learned exercises were conducted which resulted in many project templates being created and refined. Although the project reporting tool allows project managers to report against milestones, audit tests found there is no integrated project management toolset to assist in project delivery. There is the set of information requirements outlined by the Project Development Office but there is not a standard repository. IMSD reports that it has recently introduced a system architect product which may serve to store some of the necessary information. Lastly, the spreadsheet detailing some of the system development lifecycle practices is a draft and should be updated.
The regions operate with varying degrees of adherence to the business requirements process. In one case, the regional change management process was supported by an integrated tool but for the most part, there was no formal development methodology. The regions would benefit from an updated corporate toolset to allow them to better meet corporate expectations as well as to provide more consistent services across regions.
(see recommendation 4)
2.2 Risk assessment
Audit criterion: Risks associated with the business requirements and solution design should be identified, documented and analyzed.
Risks are to be assessed in several places according to the lifecycle milestone grid. Project charters and business cases are two examples of where risk assessment information should be detailed. The Project Development Office states that defining the efforts to perform environmental scans and leverage projects horizontally is a complex multi-tiered process to achieve efficiencies, and is part of that risk management process.
It was expected that branches would prepare business cases which include risk assessments to support their requests, however of the sample of projects analyzed, only some had completed a business case. Consequently, risk assessments were missing in some cases.
The end product of the Business Analysis Phase is the Detailed Business Requirements document. After a peer review of the document the IT team makes a recommendation to proceed to the next stage. The regions, on the other hand, did not have a risk assessment process to follow and would benefit from headquarters direction.
The key risks involved in application development and maintenance are characterized as environmental or organizational - not application specific; consequently an overall enterprise approach is needed for risk assessment that documents corporate and regional risks in a single architectural vision. (see recommendation 4)
2.3 Solution feasibility
Audit criterion: Studies are conducted to examine solution feasibility.
Options analysis is part of the milestone grid. With the arrival of the Investment Plan, options analysis or feasibility is often completed before the project is approved. Some branches are more mature and have this capability internally and rely on IMSD only for cost estimates. Feasibility is normally contained in formal business cases which are reviewed by IMSD.
Of the sample of projects reviewed, only some contained a high level options analysis. In some cases, this was an iterative process and options were not all assessed in a specific timeframe. For example, the options analysis for the Medical Transportation Reporting System was an ongoing process with a number of decision points throughout the project. Most regions conduct a feasibility study but there is not a set process from headquarters that makes this step mandatory. When the Branch is leading a project, there is a risk that a feasibility analysis may not occur. (see recommendation 4)
2.4 Sponsor approval
Audit criterion: The business sponsor approves and signs off on business requirements and feasibility study reports at predetermined key stages.
Sponsor signoff is included on the templates related to the milestone grid. IMSD is using the new "gating" process developed for the Investment Plan projects in which approval is required at each key step. For smaller projects, the Project Charter usually has approvals at various points: business requirement, testing, production, and end user acceptance. Signoffs are normally achieved for IMSD projects as part of the lifecycle milestone grid. In the sample of projects reviewed, some instances were found where the business owner signed off. When multiple branches are involved, the Project Sponsor handles the signoff across organizational boundaries. In the regions, any work is to be approved by the Regional Director, IMS, however there were some instances where the sign off was missing.
It is recommended that the Assistant Deputy Minister, Corporate Services Branch develop controls and guidance, in consultation with branches, for business requirements, risk assessment, solution feasibility and approval.
Management agrees with this recommendation.
The Directorate has already begun developing guidance documents to improve development of business requirements, business processes, information technology risk assessment and solution feasibility and investment planning governance and oversight.
In addition, the Directorate will seek to enhance its internal capacity to provide ongoing advice and guidance on application development and maintenance requirements. Regional realignment of information management/information technology will also contribute to enhanced rigour.
3. IT business solutions development and maintenance process
3.1 Application development process
Audit criterion: Application development and maintenance are delivered to best meet the needs of the programs and corporate priorities and adhere to the development process, accreditation, transition management and application maintenance/change management.
Applications should be developed in accordance with business requirements covering the design of the applications, the proper inclusion of application controls and security requirements, and the development and configuration in compliance with standards.
The System Development Lifecycle spreadsheet (milestone grid) outlines products, accountability and expected signoffs. The elements cover development from project selection through to business solution implementation. Since the lifecycle documentation is still a draft, project managers indicated there are negotiable elements within it and not all elements are executed by all projects. Most of the systems that were sampled had requirements documented. All systems described functionality testing and Quality Assurance and Readiness Acceptance Testing. Although the Solution Center Best Practices Workbook contains a current set of templates, no standard development toolset exists. In some cases, a test tool was used while in others, no such tool was cited. Consequently, based on the acknowledged professional body of literature for expected IT standards, Health Canada would be described as having an awareness of the processes required to develop applications but the practices are reactive and inconsistent in their implementation.
For the most part, any application development in the regions is contracted out by the programs. Where business solutions are being provided externally, business owners rely on the standards of the external service provider and consequently, regional IMS has been unable to implement departmental development standards. As a result, the development process used in the regions varied greatly. Only one region had a formal process integrated into a tool set. Regional IMS reports that they do not have accessibility or awareness of the development framework or toolsets from IMSD.
The Department would benefit from process documentation given that the development templates alone provide limited value without a formal overarching application development framework to illustrate how the templates and tools are to be used in a structured and repeatable manner.
It is recommended that the Assistant Deputy Minister, Corporate Services Branch formalize the departmental system development lifecycle.
Management agrees with this recommendation.
Various System Development Life Cycle documents already exist within the Information Management Services Directorate. These existing documents will be used to build a formalized departmental System Development Life Cycle.
Audit criterion: New systems and changes should be accredited prior to implementation, including proper testing in a dedicated environment with relevant test data, definition of rollout and migration instructions, release management and actual promotion to production, and a post-implementation review.
Accreditation and test environments are standard in IMSD and the process involves several levels of testing. The Technical Quality Assurance Team manages the movement of application assets from development to the test environment and then to the production environment. The technical team also manages the configuration of servers. Once functional testing is complete, applications are moved to the Client Acceptance Test Environment. The next step in the development process is to have the application move to the Readiness Acceptance Testing Environment for production deployment testing. The last stage is to put the application into production where the development team validates the install by running through a series of quick tests. Issues raised, either during or following a production deployment, are captured in the defect tracking system. Client sign off is required at each stage.
The accreditation process used in the regions varied. In some cases, an approach existed that included points of reference to certification and accreditation criteria checkpoints and mandatory end products. Automated test tools were not normally implemented, and test plans and cases were not formally recorded in templates. Clients often reported that testing of the application was led by the contractor without specific information requirements to ensure that clients controlled or developed test cases for functional testing purposes. The quality assurance process consists of cursory checks to ensure that the application operates, but does not include a structured set of functional tests linked to business requirements.
The process followed often depends on the project and the degree to which the client has control. In one case, although no formal phase or methodology was used for accreditation, the regional change management process served to focus delivery on acceptable application products. In some instances, the client staff would test the product and documentation with IMS staff involvement.
There was evidence of one case where the development environment existed separately from production. This situation existed because the contractor did the development and testing off-site. Only one region had an integrated tool set to manage accreditation as well as development and deployment. In cases where contractors have been engaged, often the supplier has regional approval to have direct access to the production environment at Health Canada.
Overall, accreditation and test environments are standard in headquarters while there was less internal control in the regions. In the regions, a standard approach is not consistently in use, contractors did not always produce standard documentation and test environments were not always segregated from development and production. Going forward, IMSD will need to strategize on strengthening the accreditation controls that remain within their authority as some aspects of the accreditation process are now within Shared Services Canada.
3.3 Transition management
Audit criterion: The development process requires transition planning involving the sharing of knowledge with users and IT, the provision of training to ensure the proper use and operation of applications and infrastructure, and formal signoff of the business recipient.
The transition process leading to production release includes the following documents: Release Schedule; User Manual; Release Notes; Technical Support documentation; and Production Support documentation. The value gained from deliverables produced at this stage is that they transfer knowledge and skills to enable operations and technical support staff to effectively and efficiently deliver that support.
Within the same sample of application development projects selected there was little evidence of operational or conversion procedures needed to prepare systems for deployment. In one such national application, there were explicit examples of system requirements for data conversion, yet the project information requirements did not include documentation to support the strategies, steps and procedures to execute the data conversion. The absence of these project information requirements in many instances is evidence that the templates and standards are not executed on a consistent basis. As well, in some applications that were sampled, there was no official recognition of the end of the transition phase, other than a change captured in the reporting tool. While not a standard process, lessons learned exercises are routinely used and have led to changes in the development templates resulting in some stronger controls.
Regional transition management is largely informal with the exception of one region that consistently uses an integrated tool set throughout the lifecycle of the application development process. In the remaining regions, application migration does not follow a defined management process. The migration of applications and data models is often done directly in the development or production environment without the support of migration tools or standard procedures. In some cases, a regional change management process results in IMS control over the environment. For example, change management involves weekly meetings of the senior IT resources to discuss how any change will impact the infrastructure, and what should be the "back out" plan in case of failure. Post implementation reviews in the Systems Development Life Cycle Phase are identified in the development methodology. Each review identifies lessons learned and these are recorded in a repository.
The regional applications that were sampled identified implementation components such as user manuals and rollout schedules included in deliverables from vendors. However this was offset by the lack of evidence found to ensure that the knowledge of application support components was passed on to regional IMS in the form of operations manuals or documented system specifications. In one instance, ownership of the application was not known by IMS or by the business owner. In some cases, the contractor had direct access to both the test and production environments to deliver application and other software patches and upgrades.
3.4 Application maintenance and change management
Audit criterion: All changes and maintenance relating to infrastructure and applications within the production environment should be formally logged, assessed and authorized prior to implementation and reviewed against planned outcomes following implementation.
Soon after an application is deployed, it becomes the responsibility of the maintenance group. Application maintenance goes beyond fixing defects and should consider the evolutionary development of the software and should consider the overall lifecycle costs. A basic maintenance plan should consider how users will request modifications or report problems.
Change management at IMSD relies on the Application Development Request Process.
There is usually a transition stage from one to four months once the project is released into production, whereby the development team performs sustaining activities (usually bug fixes but not enhancements). After the transition period, the project becomes an application and sustaining activities are performed by the production support team. The sustaining team manages requests for change through the Business Intake Team. Application development requests typically do not take on enhancements unless required by a departmental program due to legislative or regulatory change.
The Application Development Request Process is also considered an application maintenance support function in that it supports small enhancements. The Business Intake Team has recently been re-organized and its processes and procedures have been formalized. The development/maintenance units are working on a formal release management process, but there is not a definition yet of a "sustaining" release. Headquarters is considering mid-March and mid-September to be the planned sustaining release windows.
In the regions it was noted that an ad hoc process was followed to manage change during the deployment phase. Multiple examples were found of different processes. For example, in one region, the scope of change management items was confined to application components and parameters and did not extend to procedures or documentation. In another region, evidence showed that modifications to applications built by contractors are treated as emergency changes. The program managers emailed the contractors who applied the changes/amendments and tested on their own development platform. The contractor would then send the amendment to regional IMS with updated installation instructions. In yet another region, changes to regional applications were not controlled using IMSD's processes and contractors continue to make modifications directly in the Health Canada production environment. An instance was found where a change request received from the contractor was initially declined then approved by senior management. As mentioned in each of the other sections of the audit report, one region used the same integrated toolset to manage its change process which was the hub around which all IT modifications are managed, including application development and maintenance.
Maintenance activity is controlled and managed in IMSD however overall effectiveness and efficiency of the process is affected because of limited capacity. This impacts program areas and consequently some clients act on their own. Health Canada would benefit from a longer term IT maintenance plan outlining how the Department will manage new application priorities in conjunction with maintaining legacy systems in order to deliver the best IT services. (see recommendation 3)
C - Conclusion
Health Canada has practices to develop applications and maintain them but will benefit from strengthening the strategic direction, governance, risk management and internal controls to embed the Department's process for this activity.
Health Canada has a mandate to regulate, provide health services and develop health policy. As such, an IT architecture that draws on these common functions across the branches will aid the Department in identifying duplicate, re-usable and sharable applications. The Information Management Services Directorate is working towards developing business reference models that will align with the Department's core business lines. This direction will also provide the Department with a common nomenclature across lines of business as well as an objective platform from which the Department can choose its IT investments.
More reliance on the newly established governance and project management practices from the Investment Planning Process will provide more oversight on the IT projects as well as establishing more discipline from both the Information Management Services Directorate and the client branches for business requirements, risk assessments, option analysis and approvals via the gating process. Service level maintenance agreements for existing and new applications will provide branches with business continuity reassurance as well as having the Information Management Services Directorate develop performance metrics for maintenance.
Given that regional IT staff now report directly to the Chief Information Officer, this should strengthen practices and provide for more internal control and consistency in the delivery of the application development and maintenance services. As well, the regional staff will complement the headquarters staff from a capacity perspective but also as the two groups' bridge best practices.
Formalizing the Department's system development lifecycle methodology with updated controls and guidance will serve to strengthen the enterprise approach.
While the Information Management Services Directorate has been actively reducing the number of applications in the Department, it will need to expand this initiative to form a longer term risk-based IT plan outlining how the Department will manage new application priorities in conjunction with maintaining legacy systems in order to deliver the best IT services.
Appendix A - Lines of enquiry and audit criteria
|Criteria Title||Audit Criteria|
|Governance and Organization for Service Delivery
Sufficient oversight mechanisms exist to ensure that controls surrounding the development and maintenance of IT applications accurately reflect departmental objectives; roles and responsibilities; an appropriate level of segregation of duties; and monitoring mechanisms are established to mitigate business risks
|1. 1 Strategic direction||The Department has defined an enterprise architectural strategy for application development aligned with corporate priorities.|
|1.2 Management oversight||Appropriate oversight has been established to ensure IT applications are developed and maintained in accordance with corporate priorities.|
|1.3 Roles and responsibilities||Responsibilities and accountabilities related to IT application development and maintenance have been clearly defined, documented and understood.|
|1.4 Management reporting and monitoring||Management has established regular monitoring specific to the development and maintenance of IT applications to report on the status of: planned objectives; performance targets; solutions delivery; and client satisfaction.|
|IT Business Solutions/Needs Analysis
Health Canada develops a portfolio of IT business solutions based on an understanding of the client business environment, a determination of current and future demand and an analysis of the cost and risk of solution development/maintenance.
|2.1 Business requirements assessment||Business requirements are identified, prioritized and maintained covering the full scope of all business needs to be addressed.|
|2.2 Risk assessment||Risks associated with the business requirements and solution design are identified, documented and analyzed.|
|2.3 Solution feasibility||Studies are conducted to examine solution feasibility.|
|2.4 Sponsor approval||The business sponsor approves and signs off on business requirements and feasibility study reports at predetermined key stages.|
|IT Business Solutions Development and Maintenance Process
The development and maintenance process has been designed to best meet the needs of business users and corporate priorities within the selected cost envelope.
|3.1 Application development||Applications are developed/configured in accordance with business requirements covering the design of the applications, the proper inclusion of application controls and security requirements, and the development and configuration in compliance with standards.|
|3.2 Accreditation||New systems and changes are accredited prior to implementation, including proper testing in a dedicated environment with relevant test data, definition of rollout and migration instructions, release management and actual promotion to production, and a post-implementation review.|
|3.3 Transition management||The development process requires transition planning involving the sharing of knowledge with users and IT, the provision of training to ensure the proper use and operation of applications and infrastructure, and formal signoff of the business recipient.|
|3.4 Application maintenance and change management||All changes, including emergency maintenance and patches, relating to infrastructure and applications within the production environment should be formally logged, assessed and authorized prior to implementation and reviewed against planned outcomes following implementation.|
Appendix B - Scorecard
The rating and supporting explanation summarize the current status for each audit criterion.
|Governance and Organization for Service Delivery|
|Strategic direction||NMoI||The Information Management Services Division (IMSD) needs to establish its enterprise architecture based on a functional business process to secure the best information technology (IT) investments.||1
|Management oversight||NMoI||IMSD would benefit from using the Investment Planning Governance Framework for all application development.||2|
|Roles and responsibilities||NMoI||Roles are well defined within IMSD but there is a need to establish better communication with regional staff. A new reporting relationship for regional staff will better ensure that application development standards are being met. Service level agreements will clarify roles and responsibilities.||3|
|Management reporting and monitoring||NMiI||Performance management for application development and maintenance is tied to the Project Delivery Office and its Project Information Reporting Tool. IMSD should rely on the Investment Planning Process for additional monitoring and reporting.||2|
|IT Business Solutions/System Development Life Cycle|
|Business Requirements Assessment|
|Requirements management||NMoI||IMSD has significant components in place that represent a requirements management process. However, it will benefit from updating controls and guidance documents.||4
|Regional processes differ from region to region - a few are quite strong.||4|
|Risk assessment||NMoI||Risks are assessed in IMSD and in the regions. However, an overall approach is needed involving headquarters and regions in a single architectural vision.||4|
|There was more rigour in IMSD processes than in the regions.||4|
Options analysis and solution feasibility is a part of the milestone grid promoted by the Project Development Office. Looking at options was part of the systems in the audit sample.
|The lack of a common regional process means that options analysis is not completed in a standard way, or not done.||4|
|Sponsor approval||NMiI||Signoffs are normally achieved for IMSD projects as part of the Systems Development Lifecycle (milestone grid). More formality is needed.||4|
Signoffs are at times assumed in the regions because the sponsor often has control of the development effort.
|Development and Maintenance Process|
|Application development||NMoI||Headquarters has significant components in place that represent a development management process. The Systems Development Life Cycle is in draft form and outdated.||5|
|Regional processes differ from region to region - a few are strong, however none relate to what headquarters is currently using.||5|
Accreditation and test environments are standard in headquarters. Some parts of accreditation are now under the control of the Shared Services Canada Agency.
There was less control in the regions - a standard approach is not always in use, contractors did not always produce standard documentation and test environments were not always segregated from development and production.
Control over transition was relatively strong in headquarters.
There was less control in the regions - both because they did not use a standard approach, and because their relationship with their key client branch leads to conflict. A gap exists in regions where suppliers often have direct access to production environments.
|Application maintenance and change management||S||
Maintenance activity is controlled and managed in headquarters. Effectiveness of the process is affected by a capacity concern which in turn has an impact on client reactions.
Regional changes can be affected by the degree to which clients understand their requirements (see requirements management) and the degree to which clients feel they can act on their own.
- - Satisfactory
- - Needs minor improvements
- - Needs moderate improvements
- - Needs improvement
- - Unsatisfactory
- - Unknown; Cannot be measured
Appendix C - Application delivery framework
This diagram portrays the framework in Health Canada supporting the development and maintenance of application software. The scope of this audit focused on the application development and maintenance processes (un-shaded section) and did not include the planning processes through which projects are selected (shaded section).
- Date modified: