Public Services and Procurement Canada
Evaluation of Public Services and Procurement Canada’s use of selected consulting services

Consulting service contracts are used to provide the Department with specialized knowledge, expertise and advice, on a cost-considered basis, to inform the planning and conduct of activities and to support decision making across PSPC Branches and Regions. This report presents an overview of a department-initiated evaluation of relevance, performance and cost-effectiveness intended to inform decision making regarding use going forward of a subset of service contract types used by PSPC in the area of consulting services: Management Consulting, Information and Technology Consulting, and Accounting and Audit Services.

Table 1: Management action plan for recommendation 1
Table 2: Management action plan for recommendation 2
Table 3: Management action plan for recommendation 3
Table 4: Management action plan for recommendation 4
Table 5: Identified files by service type and approach (competitive or non-competitive) – Source: SIGMA, March 2021
Table 6: Identified files by service type (Source: SIGMA, March 2021)
Table 7: Limitations encountered by the evaluation in the execution of the planned methods and the actions taken by the evaluation to mitigate the impacts of the limitations

Introduction

The use of consulting services contracts is one of several strategies employed by Public Services and Procurement Canada (PSPC) as it works to deliver high-quality, central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions. While PSPC is commonly known for establishing contracts for other departments and agencies, it also establishes contracts for its own use: This evaluation addresses a sub-set of contracts established for use by PSPC in the area of consulting services, and is as such titled Evaluation PSPC’s Use of Selected Consulting Services. Consulting service contracts are used to provide the Department with specialized knowledge, expertise and advice, on a cost-considered basis, to inform the planning and conduct of activities and to support decision making across PSPC Branches and Regions.

For the period from October 1, 2017, to August 31, 2020, the department spent approximately $7.5 Billion^{Footnote 1} on 14,765 new service contracts over $10,000 in value in the execution of its core responsibilities to support the achievement of departmental objectives. Service contracts are put in place after a competitive process, while in certain circumstances these types of contracts are established using non-competitive methods.

This report presents an overview of a department-initiated evaluation of the relevance, performance and cost-effectiveness of the use of a subset of service contract types by PSPC, intended to inform decision making regarding their use going forward. To this end, the evaluation identified 734 contracts with a total value of approximately $425 million^{Footnote 2}. These 734 contracts represents approximately 6% of the total number of new service contracts established by PSPC valued over $10,000 during this period and roughly 6% of the total value. The service areas selected for consideration were:

management consulting: Consulting services for financial management, transportation, economic development, environmental planning, public consultation and other consulting services
information and technology consulting: Professional contracting and consulting services that support any information technology service, including distributed computing, application and database development and maintenance, production and operations computing, telecommunications, IT security and IT program management services
accounting and audit services: Accounting and audit services including financial administration and bookkeeping services, financial control, accounting and audit services, budget control systems, tax studies and other financial and related services

About this evaluation

The evaluation was conducted in accordance with the Treasury Board Policy on Results and Directive on Results for the Government of Canada, executed by the Evaluation Services Directorate in fiscal year 2021 to 2022 under the authority of the Head of Evaluation for PSPC with guidance from the PSPC Evaluation Advisory Committee.

All recommendations made in this report have been accepted by the Department. PSPC’s management action plan in response to the findings of this evaluation is presented in Annex 1 of this report.

Evaluation scope

The consulting service contracts considered as part of this evaluation were established for use within PSPC and aligned to following criteria:

valued over $10,000
established by PSPC in the period between October 1, 2017, and August 31, 2020; and
representing the following types of service:
- Management Consulting
- Information Technology and Telecommunications Consulting, and
- Accounting and Audit Services

Descriptions of these services (as well as corresponding coding) were taken from the Government-wide chart of accounts for Canada, which is a framework that prescribes the way departments and agencies should identify, collect and report financial transactions including contracts.

Excluded from the scope were:

contracts for goods and construction, as well as services related to goods, construction, and property maintenance (for example, architecture and engineering services and building design services)
contracts prepared by the Acquisitions Program on behalf of other government departments and agencies
service contracts coded as “other” in the department’s financial management system (SIGMA)
project management services
training and coaching services
services recently subjected to an evaluation (for example, fairness monitoring)

Applying the above criteria, the evaluation identified 734 new contracts valued over $10,000 for management consulting, Information Technology and telecommunications consulting, and accounting and audit services. Summary descriptions of contract file characteristics are provided in Annex 2. The contracts established could be competitive (where there process includes the solicitation of bids from suppliers) or non-competitive (where the process establishes a contract with a preselected contractor under justifiable circumstances). At the time of this evaluation, contracts could be managed by different administrators within the department, depending on its nature.

Evaluation objective

The objective of the evaluation was to assess the relevance, performance and cost-effectiveness of PSPC’s use of selected consulting services.

Evaluation approach

The evaluation used the following lines of evidence:

document review – Program documentation, related assessments and procurement policy and training documents were reviewed
administrative data – Contract data was extracted from the contracting component of the department’s financial management system (SIGMA) in March 2021. Based on this data set, the evaluation identified 734 contracts valued at over $10,000 as having been established for the 3 selected services during the period under review
contract file review – Of the 734 contracts identified, 105 contract files were screened as part of the random sample selection process
- of these, 38 files were used for the detailed contract file review
- overall, these files are representative of different service types, branches/regions, price ranges and competitive/non-competitive contracts with an increased representation of the Information Technology/Telecommunications consulting service type (at the request of PSPC senior management)
surveys – 2 different surveys were conducted at the director, manager and working level: one with program officials (meaning that, business owners using of the select consulting services) and the other with contracting officials (meaning that, those who would support the set up of the contracts requested by business owners). Respondents were to answer questions based on specific contracts from the file review sample
interview – 19 interviews were conducted with Assistant Deputy Ministers (ADM), Associate Assistant Deputy Ministers (AADM), Regional Director General (RDGs), Director Generals (DGs) and Senior Directors

Issues related to the accuracy of administrative data and the incompleteness of some contract files (notably those screened in for the contract file review) had an impact on the Evaluation Services' ability to conduct a complete analysis. As with all evaluations, limitations stemming from the data collection methods were mitigated to the extent possible by the fact that the findings were from, and are presented here, multiple lines of evidence. A discussion of the specific limitations that impacted the evaluation and the mitigation measures taken to address them can be found in Annex 3.

Key findings

The findings from this evaluation are presented under the issues selected for this project: Relevance, Performance, and Cost-effectiveness.

Relevance

The selected consulting service areas of management consulting, IT and telecommunications consulting and accounting and audit services are viewed by PSPC management as helpful in meeting their operational obligations, particularly for responding to a need for specialized or impartial support or to quickly respond to internal resource capacity deficits.

Reasons for using select consulting services

Based on the contract file reviews, surveys and interviews, the most common reasons for using management consulting, IT and telecommunications consulting and accounting and audit services at PSPC were:

the need for specialized skills, knowledge or expertise that were unavailable in house
insufficient internal resources
the need for independence and impartiality

Using the selected consulting services due to a lack of specialized skills, knowledge and expertise was by far the most common reason identified, accounting for 50% of reasons drawn from contract file reviews and 59% of reasons noted in survey responses. Moreover, 100% of senior managers interviewed mentioned that this was the main reason their organizations used the selected consulting services. Other common rationales that were identified through the file reviews, surveys and interviews^{Footnote 3} included:

needing flexibility to handle spikes in demand
being able to access the national and international networks of suppliers and various sectors for benchmarking purposes
being able to meet resource needs more easily and more quickly than the internal staffing process
using consultants as part of an overall operational or business strategy to supplement internal resources
needing to meet legal, policy, regulatory, professional and ethics requirements

Performance

The quality of the deliverables received is viewed by users as sufficient and management reports effective strategies are in place to manage the performance of consultants. Moreover, the expertise obtained from consulting services is being shared within the department and is contributing to decision-making at various levels.

Quality of deliverables

Survey, Interview and file review data suggest that deliverables received by PSPC were satisfactory. When asked if deliverables received were of sufficient quality, 80% of those surveyed provided a positive response (74% of respondents noted that all were of sufficient quality and 6% of respondents noted that most were of sufficient quality), 16% of respondents could not recall and 3% of respondents noted most deliverables were not of sufficient quality. Where it was noted that most deliverables were not of sufficient quality, comments left for this category selection noted that the quality of work was poor due to a lack of collaboration with the consultant. Furthermore, approximately 70% of interviewed senior managers believed that most of the deliverables obtained from these contracts were of sufficient quality, while approximately 30% believed that some deliverables were of sufficient quality. Where it was noted that some (rather than all) deliverables were of sufficient quality, comments remarked that a ‘mixed-bag’ of deliverables can be due to limitations on the part of both the contractor (such as in scope setting) and the consultant (such as capability). Finally, based on where information was available through the contract file review^{Footnote 4}, the evaluation noted that deliverables met all quality requirements in 76% of instances. Moreover, evidence of formal acceptance of deliverables (whether in the form of approved invoices or documented attestations of receipt and acceptance) was found in 100% of the contract review files.

Management of consultants

74% survey respondents believe the strategies employed to manage the performance of consultants were successful (while noting that 19% of respondents did not have an opinion on this matter). Reported strategies used to manage consultants include:

ensuring expectations were clearly defined in the contract terms and conditions or the Statement of Work (SOW)
reviewing early drafts of deliverables
providing ongoing feedback
frequently touching base with consultants regarding schedules/deliverables
linking payments to the receipt of acceptable products

The risk of developing employer-employee relationships decreases when contracts are for specific deliverables and have clear delivery dates. Survey, contract file review and interviews demonstrate that this risk is being managed:

55% of survey respondents indicated that contracts were awarded for specific deliverables with clear delivery dates
86% of contracts selected for file review were for specific deliverables that were to be submitted within a defined timeframe
78% of interviewees indicated that contracts established by their organizations were generally for specific deliverables with clearly established delivery dates

Information use, sharing and impact

Document review, survey and interview data confirms that there is use and sharing of information from consulting contracts among departmental teams.

Through the review of program and organizational documents, the evaluation determined that information obtained from 37% of the reviewed contracts was included in plans, decks and reports prepared within the department. In addition, 46% of survey respondents indicated that information obtained from consulting contracts was included in planning documents, decks and reports at the program, sector, branch and/or departmental level.

The evaluation also found that expertise obtained from these contracts is being shared with committees and working groups at various levels throughout the department. Approximately 43% of survey respondents indicated that information from contracts was shared with working groups or committees at either the sector level, directorate level, branch level, departmental level or interdepartmental level. When asked about the extent to which the information shared with working groups and committees was useful to the work of these bodies, 67% of those who responded indicated that it was very useful while 23% found it somewhat or moderately useful.

Furthermore, 100% of the senior managers interviewed agreed that information from the use of select consulting services was being considered as part of departmental decision-making. In addition, 75% of survey respondents indicated that information obtained from consulting services was used in decision-making related to operational, program delivery, while 57% indicated that it was used in decision-making related to program and organizational planning and design.

Cost-effectiveness

While value for money and cost-effectiveness is perceived by the majority of consulting service users, the evaluation notes that in some instances insufficient preparation and planning may have limited the cost-effectiveness of the use of select consulting services.

Perspectives on value for money / cost-effectiveness

Program officials and management frequently agree that the vendor selected represented the best value for the work completed. The evaluation found that 64% of survey respondents agreed that the selection criteria used for their contracts resulted in a vendor representing the best value for money, while 36% did not know. No survey respondents replied that the vendor selected did not represent the best value for money. Furthermore, of the 12 interviewees who responded to the question of cost-effectiveness, 83% indicated that expertise is generally provided in a cost-effective manner, while the other 17% indicated that cost-effectiveness can vary by contract.

Timeliness of deliverables

With regards to timeliness, while survey results found that 68% of responding participants noted that most or all of their contract deliverables were received in a timely manner, the contract file review noted that all deliverables were provided on time in approximately 55% of cases.

The Treasury Board Contracting Policy defines amendments as “an agreed addition to, deletion from, correction or modification of a contract.” Amendments can be made to a contract’s scope, timeframe, value, or a combination thereof. The amendment process can be initiated by the client, the contractor, or the contracting authority. When an amendment request is initiated by the client or the contractor, the contracting authority is responsible for reviewing the amendment request and rationale in order to ensure that falls within the scope of the applicable procurement policy, with guidance from internal policy and legal stakeholders as needed. The evaluation noted that approximately 13% of the 734 identified contracts had undergone multiple amendments. Additionally, the contract file review noted that approximately half (17/38) had undergone an amendment. Further details on the use of amendments can be found in Annex 2. These findings provide a limited context for understanding the use of amendments as part of the contracting for select consulting services, as it was out of the scope of this evaluation to complete a comparative analysis against an external benchmark or an accepted norm/target for this management activity. Further analysis on the use of amendments in the context of strengthening cost effectiveness in the use of select consulting services may be an area for future consideration by the Department.

Where rationales for amendments in sample contract files were available, they reflected causes such as:

work taking longer than expected
changes in supplier resources
changes in delivery dates and scope
COVID-19 related delays

Survey respondents provided additional reasons for why amendments may be needed, including:

vendor level of effort greater than expected
COVID-19 delays and lengthy procurement process
delays in consultants obtaining access to essential data information and key internal resources
changes in scope

While the rationales provided for contract revisions include elements outside of the influence of the department, such as the unprecedented impact of the Covid-19 pandemic, the information collected by the evaluation suggests that insufficient preparation and planning in some instances may have limited the cost-effectiveness of the use of select consulting services through the need for changes in time, value and scope of contracts.

Other findings

This section provides an overview on key additional findings resulting from the conduct of this evaluation with significance to departmental operations, notably that:

there is a lack of consensus on what is the best use of external resources
while not a requirement, there was limited formal consideration of other human resourcing options prior to going forward with the use of the select consulting services
the evaluation encountered limitations in the coding and completeness of contract files which impeded its ability to effectively and efficiently review files relevant to the scope of the engagement and may indicate risks to the department’s identification, collection and reporting on contracting

Use of external resources

Opinions on whether there is an overreliance on external consulting services were evenly split among the senior management members interviewed for this evaluation. There were interviewees who noted that their use and need for select consulting services, particularly in areas of highly specialized skill sets or where the need for impartiality (such as expert witness statements), was long standing. These findings provide a limited context for understanding how the select consulting services have been used within PSPC, as it was out of the scope of this evaluation to complete a comparative analysis against an external benchmark or an accepted norm/target for this management activity. The question of what constitutes appropriate use of the select consulting services may be an area for future consideration by the Department.

Consideration of other human resourcing options

The contract for services reviewed by the evaluation were completed under the direction of the Contracting Policy^{Footnote 5} of the Government of Canada. The objective of the Policy is to “acquire goods and services and to carry out construction in a manner that enhances access, competition and fairness and results in best value or, if appropriate, the optimal balance of overall benefits to the Crown and the Canadian people” and identifies a series of requirements that contracting authorities must adhere to when contracting for services. These policy requirements did not include formally reviewing the possibility of other human resourcing options to conduct the work required. While consideration of other human resourcing options was not a requirement at the time these contracts were pursued, the evaluation considered awareness and use of alternative human resource options as part of its objective to help inform decisions regarding the department’s use of these services in future years.

While the evaluation found that insufficient internal resources was one of the most common reasons provided for seeking a contract for services, evidence collected as part of this evaluation presented a low level of awareness and consideration by departmental teams regarding the use of other human resource options such as term, casual, deployment, assignment and/or secondment opportunities as an alternative to contracting for services. Some interviewees (16%) spoke to exploring internal (term, casual, or student) or external (academia) human resourcing options prior to pursuing contracting for services. In comparison, more survey respondents than interviewees noted that other human resourcing options were considered before contracting, with 27% of respondents identifying one or more possibilities having been considered but not pursued (such as term, casual, student and/or deployment/assignment staffing), however, most survey respondents replied that no other staffing options were explored (41%) or replied that they did not know if other human resourcing options were considered before contracting for a service (32%). Also, when asked whether any plans were currently in place to hire or develop staff with the capacities and skills being sought, 38% of survey respondents replying to this question answered no.

These findings suggest that there is an opportunity to raise awareness on and encourage the exploration of traditional and innovative human resourcing alternatives to provide business owners with additional options to meet short-term resource requirements. To this end, the evaluation has made the following recommendation:

Recommendation 1: It is recommended that the department raise awareness on existing Human Resource (HR) options so they can be taken into consideration prior to using selected consulting services. It is also recommended that a set of guiding principles and criteria be developed and distributed to inform decisions on whether or not to use selected consulting services.

Availability of information

The evaluation noted limitations in the coding and completeness of contract files which impeded its ability to effectively and efficiently review files relevant to the scope of the engagement. A discussion on the limitation posed to the evaluation overall by the identified data entry errors and contract file incompleteness, and mitigation measures taken, is found in Annex 3 of this report.

Further to its impact on the conduct of the evaluation, the issues encountered may indicate risks to the department’s identification, collection and reporting on contracting. The challenges encountered by the evaluation’s data collection are areas for improvement by the department in support of on-going utility and quality of the information captured within SIGMA and overall record management.

Contract files

In terms of the completeness of contract files, approximately 20% of the 105 contracts that were screened as part of the sample selection process had to be excluded from further consideration because key documents could not be provided to the evaluation team within the timeframe of the evaluation. In consideration of the challenges faced in accessing and consolidating full contracting information from across various areas of the department, the evaluation has made the following recommendation:

Recommendation 2: It is recommended that the department implement a strategy to ensure the completeness and accessibility of contract files to facilitate contract management, monitoring and reporting activities.

Data coding

The evaluation found that approximately 3,127 contracts or 23.92% of all service contracts over $10,000 established within the period under review were coded as falling within the miscellaneous accounting categories of services such as “other professional services not elsewhere specified” within the department’s financial administration system (SIGMA). Given the high number of contracts coded to this category, there is a need for further analysis to ensure that the category does not incorrectly include contracts that should have been further coded to a specific service sub-type such as those reviewed as part of this evaluation (Management Consulting, Information Technology and Telecommunications Consulting; and Accounting and Audit Services). There is also a need for guidance for those entering contract information in SIGMA so that these miscellaneous categories are only used when needed.

Additionally, the file sample selection process and the detailed file review led to the identification of inaccuracies in the information recorded in SIGMA within the areas of contract and/or amendment values; contract numbers; contract creation dates; and contract files that were created in error.

As such, the evaluation has made the following recommendations:

Recommendation 3: It is recommended that clearer guidance on how to correctly input contract- and account-related information should be developed and disseminated in order to address data entry errors in SIGMA.

Recommendation 4: It is recommended that the department conduct a detailed review on contracts captured within miscellaneous categories or implement a process to ensure these categories are only used when necessary, so as to improve data specificity and quality for analysis and decision-making.

Evaluation summary

This report presents an overview of a department-initiated evaluation of the relevance, performance and cost-effectiveness of the use of a subset of service contract types by PSPC intended to inform decision making regarding their use going forward. To this end, the evaluation identified 734 contracts with a total value of approximately $425 million. These 734 contracts represents approximately 6% of the total number of new service contracts established by PSPC valued over $10,000, and roughly 6% of their total value, in the period between October 1, 2017, and August 31, 2020. The service areas selected for consideration were Management Consulting; Information and Technology Consulting; and Accounting and Audit Services.

The evaluation was conducted in-house under the oversight of the PSPC Head of Evaluation. Methods used for this evaluation were surveys, interviews, administrative data review and a detailed review of 38 contract files.

The selected consulting service areas are viewed by PSPC management as helpful in meeting their operational obligations, particularly for responding to a need for specialized or impartial support or to quickly respond to internal resource capacity deficits. Overall, the evidence suggests that business owners are satisfied with the quality of deliverables and that effective strategies are in place to manage the performance of consultants. Moreover, the expertise obtained is being shared within the department and is contributing to decision-making at various levels. Users report value for money is being obtained from these contracted services, however, the evaluation found that in some instances that insufficient preparation and planning may have limited the cost-effectiveness of the use of select consulting services by impacting the timeliness of deliverables.

The evaluation included additional observations arising from its conduct with significance to departmental operations. The evaluation noted that there is a lack of consensus on what is the best use of external resources. While not a contracting requirement at the time of the evaluation’s conduct, there was limited documentation on the consideration of other human resourcing options prior to going forward with the use of the select consulting services. As well, the evaluation encountered limitations in the coding and completeness of contract files which impeded its ability to effectively and efficiently review relevant files which may indicate risks to the department’s identification, collection and reporting on contracts. The additional observations from this evaluation led to the following 4 recommendations, which have been accepted by the Department and for which management action plans have been established:

Recommendation 1: It is recommended that the department raise awareness on existing HR options so they can be taken into consideration prior to using selected consulting services. It is also recommended that a set of guiding principles and criteria be developed and distributed to inform decisions on whether or not to use selected consulting services.

Annex 1: Management action plan

Recommendation 1

It is recommended that the department raise awareness on existing HR options so they can be taken into consideration prior to using selected consulting services. It is also recommended that a set of guiding principles and criteria be developed and distributed to inform decisions on whether or not to use selected consulting services.

Office of primary interest: Human Resources Branch

Office of secondary interest: Procurement Branch (for management response 1.2 only)

Table 1: Management action plan for recommendation 1
Management action plan – Evaluation recommendation 1	Target date	Status as of May 2024
1.1 Human Resources Branch will organize, twice a year, information sessions for managers to discuss available staffing and recruitment options (including short-term ones)	October 2022, March 2023	Implemented January 2024
1.2 Human Resources Branch will consult colleagues in Procurement Branch and work with them to develop a 1-pager to inform managers of potential recruitment options available to them (including short-term ones)	January 2023	Implemented December 2023

Recommendation 2

It is recommended that the department implement a strategy to ensure the completeness and accessibility of contract files to facilitate contract management, monitoring and reporting activities.

Office of primary interest: Procurement Branch

Office of secondary interest: All PSPC Branches and Regions

Table 2: Management action plan for recommendation 2
Management action plan – Evaluation recommendation 2	Target date	Status as of May 2024
2.1 Procurement Branch will develop a department-wide communique regarding procurement documentation procedures for contracting authorities and project authorities as currently outlined in the Public Services and Procurement Canada Contract management guide, including a checklist	August 31, 2022	Revised target date: September 2024
2.2 Procurement Branch will offer an informational session regarding procurement documentation procedures on an annual basis via PSPC’s Integrated Forum on Procurement and Materiel Management	November 30, 2022	Implemented November 2022
2.3 Review the PSPC Contract Management Guide content with respect to procurement documentation procedures, looking for opportunities to strengthen existing guidance or to reflect recent policy developments	June 1, 2023	Implemented April 2024

Recommendation 3

It is recommended that clearer guidance on how to correctly input contract- and account-related information should be developed and disseminated in order to address data entry errors in SIGMA.

Office of primary interest: Finance Branch. It will continue and/or enhance existing activities to monitor the quality of general ledger integrity for expenditures recorded in the department’s financial system

Office of secondary interest: Procurement Branch (for management response 3.4 only)

Table 3: Management action plan for recommendation 3
Management action plan – Evaluation recommendation 3	Target date	Status as of May 2024
3.1 Finance Branch is engaged to collaborate with the In-Service Support User Training team within SIGMA to leverage the technical training documentation on the Chart of Accounts and incorporate more functional content to raise awareness	March 2023	Revised Target Date: September 2024
3.2 Finance Branch will develop and disseminate 3 separate financial training curricula for end users in SIGMA, Financial Analysts and delegated budget managers to provide guidance on financial coding and data entry	March 2023	Revised Target Date: September 2024
3.3 Finance Branch will prepare a communique informing SIGMA users that the sector is working on improvements to reduce known common errors made by users in contract data entry, and urging them to make every effort to ensure the accuracy of the codes they input in the financial system	November 2022	Implemented March 2023
3.4 Finance Branch will review, update and publish SIGMA contract/account-related work instructions on the intranet supported by the Procurement Branch’s Departmental Acquisitions Services (DAS) Directorate in alignment with their upcoming “business guide”	March 2023	Implemented March 2023
3.5 Finance Branch will review SIGMA contract/account-related training in order to identify possible improvements, including consideration for additional information regarding the chart of accounts/General Ledger and for the participation of business expert/subject-matter expert to training sessions through: list of possible value-added improvements on SIGMA contract/account-related training if applicable, updated SIGMA contract/account-related training	A: December 2022 B: March 2023	Implemented June 2023

Recommendation 4

It is recommended that the department conduct a detailed review on contracts captured within miscellaneous categories or implement a process to ensure these categories are only used when necessary, so as to improve data specificity and quality for analysis and decision-making.

Office of primary interest: Finance Branch

Office of secondary interest: None

Table 4: Management action plan for recommendation 4
Management action plan – Evaluation recommendation 4	Target date	Status as of May 2024
4.1 Financial Operations Sector (FOS) will review the mapping of professional and special services General Ledger accounts to economic objects (ECON) in order to determine whether mappings to the ECON “other professional services not elsewhere specified” can be reduced and attributed elsewhere in an effort to more accurately identify the nature of these expenditures	December 31, 2022	Implemented January 2023
4.2 Finance Branch will undertake the timely communication of General Ledger changes via posting updated version of General Ledger Accounts Mapping Structure on Finance Branch intranet site	Semi-annually	Implemented January 2023
4.3 Finance Branch will continue to provide quarterly reports on verification results, including administrative errors where incorrect general ledger coding is identified. In addition, the National Account Verification Program may conduct targeted sampling depending on the recurrence of these administrative errors	March 2023	Implemented March 2023
4.4 Finance Branch Financial management advisors will implement a process to reinforce the importance of financial coding for data integrity (including limited use of miscellaneous consulting services) during their Departmental Management Report meetings with clients	July 2022	Implemented June 2023
4.5 As part of the year-end Public Accounts reporting on professional and special services, Finance Branch performs a variance analysis on all reporting and follows up with Financial and Managerial Accounting System as required to seek explanations for large/unusual variances in order to identify potential coding errors	Annually	Implemented October 2022

Annex 2: Select contract file characteristics

The evaluation completed analysis on 734 extracted contract files identified as within scope for this project in order to better understand their characteristics. Furthermore, 38 contract files underwent detailed analysis to support the evaluation. The following descriptions are provided to inform the reader’s understanding of the selected sub-set of contracts included in this evaluation. Given a lack of precision in identifying the population of relevant contracts, and the limited time period of the evaluation’s scope, normative conclusions cannot be drawn from the information available.

Extracted contract file characteristics (734 files)

Table 5: Identified files by service type and approach (competitive or non-competitive)^{table 5 note *}
Services	Competitive units	Non-competitive units	Total by service type
Management consulting	512	112	624
IT and telecommunications consulting	71	29	100
Accounting and audit services	7	3	10
Total	590	144	734
Table 5 Notes Table 5 Note * Source: SIGMA, March 2021 Return to table 5 note * referrer

Number of contracts

As presented in Table 1, the highest number of service types was management consulting, equalling 85% of the total number of contracts established for the 3 selected services. IT and telecommunications accounted for approximately 13% of the total number of contracts while accounting and audit services accounted for less than 1%.

Contract type (Competitive / Non-competitive)

As presented in Table 1, non-competitive contracts accounted for less than 20% (n=144) of all contracts. Of the non-competitive variety, the distribution among service types was: 78% of management consulting, 20% of Information Technology and Telecommunication and 2% of accounting and audit services.

Contract users

The 3 Branches/Regions that established the most contracts for the 3 service types were:

Real Property Services
Departmental Oversight Branch, and
Ontario Region

Values

As noted earlier in this evaluation, the total value of identified 734 evaluations is approximately $425 million. Management consulting services contracts accounted for approximately 64% of their total value. Information Technology and telecommunications accounting for approximately 35% of their total value. Information Technology and Telecommunication accounted for the remaining approximate 1% of total value.

The values of contracts for the 3 service types ranged from $10k to $ 50M, noting that the evaluation did not analyze the contract durations or scope of work.

Amendments

Of the 734 contracts, approximately 45% of the 734 contracts were amended once and 13% amended more than twice.

The total value of amendments put in place within the period under review was approximately $27.4 million and is included in the total figure of approximately $425 million calculated for the 734 contracts.

Contract file review characteristics (38 files)

Table 6: Identified files by service type^{table 6 note **}
Services	Number of files included in the detailed review
Management consulting	28
IT and telecommunications consulting	8
Accounting and audit services	2
Total	38
Table 6 Notes Table 6 Note Source: SIGMA, March 2021 Return to table 6 note referrer

Frequency

As presented in Table 2, the most frequently found type of service was management consulting, equalling 74% of the total number of contracts reviewed in-depth. IT and telecommunications accounted for approximately 21% of the total number of contracts while accounting and audit services accounted for 5%.

Contract type (Competitive / Non-competitive)

From the 38 contract files, 71% (n=27/38) were awarded through a competitive assessment and 29% (n=11/38) through a non-competitive one. 37% (n=14/38) were multi-year contract files. Of these multi-year contracts, 21% (n=3/14) were open contract in a “need basis”. 93% (n=13/14) of the multi-year contract files went through a competitive process.

Contract users

The 3 Branches/Regions that established the most contracts were:

Real Property Services
Departmental Oversight Branch, and
Acquisition Branch

Values

The contract range for the 38 contract files were distributed as follows:

18 of the contract files were between $10k-$40k
13 of the contract files were between $40k-$250k
4 of the contract files were between $250k-$1M
3 of the contract files were from $1M to above

Amendments

Of the total 38 contracts, 17 had at least one amendment while 21 did not have amendments:

7 contracts were amended for time only
8 contracts were amended for time and value
1 contract was amended for time, value and scope
1 contract was amended for value only

Annex 3: Evaluation methods by approach, limitations and mitigation activities

Table 7: Limitations encountered by the evaluation in the execution of the planned methods and the actions taken by the evaluation to mitigate the impacts of the limitations
Method	Approach	Limitations	Mitigation activities
Contract file review (including SIGMA data)	Data sets extracted from the contract management component of SIGMA in March 2021 in order to identify relevant contract files for the period under review 40 contract files were to be subjected to a detailed file review, drawn from a sample of 105 contracts Sampling of contracts was to be diverse and proportionally representative of distribution among 3 service types; Branches/Regions; Price ranges; Competitive/non-competitive process Some oversampling was done to reflect areas of interest identified by senior management (for example, IT and Telecommunications Consulting)	The evaluation identified 734 files relevant to the scope of this evaluation, but has not referred to it as the population: Some contracts falling within scope may have been omitted from the random selection process as they may have been coded under “other” services rather than one of the 3 selected consulting types Overall, the evaluation had to exclude approximately 50% of the sampled 105 files due to information quality and completeness issues Contract files are retained in multiple areas of the department Miscoding of contracts, unavailability/ incompleteness of files and evaluation timelines resulted in: sample size being reduced to 38 with changes to original sample distribution In order to ensure at least 38 files could be retained for a detailed, some contracts had to be retained even though their files were incomplete. For example, 3 contracts were missing task authorizations and 2 were missing amendment information Inaccuracies in SIGMA data were identified only for contracts that were screened as part of the random sample selection process or subjected to a file review	Decreased sample size has no impact on the confidence interval (8.15%) or margin of error (90%) When resampling, every effort was made to maintain the diversity and proportional representation of the new sample Information gaps were filled by conducting research in SIGMA and contacting employees involved in the creation/ management of contracts Limitations in the information derived from the SIGMA data and contract files were addressed when establishing findings through the use of multiple lines of evidence (triangulation) when assessing the evaluation issues The evaluation has proposed recommendations to the department to support further mitigation in this area
Surveys	Overall response rate: 67% (59% for Program Officials; 82% for Contracting Officials) Survey Completion rate: 39% for Program Officials, 68% for Contracting Officials Given the incomplete response rate, survey results provided supplementary information on 21 contracts while information was not collected on 17 contracts included in the detailed review	Overall response rate: 67% (59% for Program Officials; 82% for Contracting Officials) Survey Completion rate: 39% for Program Officials, 68% for Contracting Officials Given the incomplete response rate, survey results provided supplementary information on 21 contracts while information was not collected on 17 contracts included in the detailed review	Potential distortions and low completion rates were taken into consideration when establishing findings through the use of multiple lines of evidence (triangulation) when assessing the evaluation issues
Interviews	2 questionnaires prepared for 2 stakeholder groups: (Associate) Assistant Deputy Ministers and Regional Director Generals, and Director Generals and Senior Directors A total of 19 interviewees (11 (Associate) Assistant Deputy Ministers and 8 Director Generals/Senior Directors)	Given the nature of verbal interviews, information varied in scope, depth and breath	Interview notes were validated by respondents Differences in depth and bread of qualitative information received was taken into consideration when establishing findings through the use of multiple lines of evidence (triangulation) when assessing the evaluation issues
Document review	The evaluation collected and reviewed procurement policy and training documents; program documentation and related assessments	Complete documentation may not have been fully retrieved	Every effort was made to collect complete and relevant documentation, with any limitations consideration when establishing findings through the use of multiple lines of evidence (triangulation) when assessing the evaluation issues

Footnotes

Footnote 1

Extracted from the total value of original contracts plus the total value of their subsequent amendments that were awarded within the same period.

Return to footnote1 referrer

Footnote 2

Extracted from the original value of these contracts, and any subsequent amendments, established within this same period.

Return to footnote2 referrer

Footnote 3

Individual survey and interview respondents may have provided more than one reason for using the selected consulting services.

Return to footnote3 referrer

Footnote 4

In some cases, no deliverables were specified in the contracts or there was no documentation available as to the assessment of deliverable quality.

Return to footnote4 referrer

Footnote 5

The Contracting Policy was rescinded on May 13, 2022. It has been replaced by the procurement requirements in the Policy on the Planning and Management of Investments and the Directive on the Management of Procurement.

Return to footnote5 referrer

Page details

Date modified:: 2024-07-10

Language selection

Search

Public Services and Procurement Canada
Evaluation of Public Services and Procurement Canada’s use of selected consulting services

On this page

Recommendation 2

Recommendation 3

Recommendation 4

Annex 2: Select contract file characteristics

Extracted contract file characteristics (734 files)

Table 5 Notes

Number of contracts

Contract type (Competitive / Non-competitive)

Contract users

Values

Amendments

Contract file review characteristics (38 files)

Table 6 Notes

Frequency

Contract type (Competitive / Non-competitive)

Contract users

Values

Amendments

Annex 3: Evaluation methods by approach, limitations and mitigation activities

Page details

Public Services and Procurement CanadaEvaluation of Public Services and Procurement Canada’s use of selected consulting services

On this page

Recommendation 2

Recommendation 3

Recommendation 4

Annex 2: Select contract file characteristics

Extracted contract file characteristics (734 files)

Table 5 Notes

Number of contracts

Contract type (Competitive / Non-competitive)

Contract users

Values

Amendments

Contract file review characteristics (38 files)

Table 6 Notes

Frequency

Contract type (Competitive / Non-competitive)

Contract users

Values

Amendments

Annex 3: Evaluation methods by approach, limitations and mitigation activities

Page details

Public Services and Procurement Canada
Evaluation of Public Services and Procurement Canada’s use of selected consulting services