Evaluation of the Certification Program for Procurement and Materiel Management Communities in the Government of Canada
Attention: This Program is ending on March 31, 2023. No new participants are being accepted. With the support of partners and the Canada School of Public Service, TBS is leading a new professional development framework and curriculum for the procurement and materiel management communities. Stay tuned!
On this page
- Introduction
- Results at a glance
- About the certification program
- Evaluation methodology and scope
- Limitations of the evaluation
- Relevance
- Performance
- Immediate outcomes
- Intermediate outcomes
- Alternatives to the current program design
- Recommendations
- Appendix A: journey to become a certified federal specialist in procurement and materiel management and related evaluation findings
- Appendix B: logic model for the certification program
- Appendix C: costs of the certification program and key assumptions for costing
- Appendix D: evaluation methodology
- Appendix E: 2014 and 2018 competencies
- Appendix F: governmental procurement and materiel management training programs in other countries
- Appendix G: Management response and action plan
Introduction
This document presents the results of an evaluation of the Certification Program for Procurement and Materiel Management Communities in the Government of Canada, which is managed by the Acquired Services and Assets Sector Communities Management Office within the Office of the Comptroller General (OCG) at the Treasury Board of Canada Secretariat (TBS). The evaluation was carried out by the TBS Internal Audit and Evaluation Bureau (IAEB), with the assistance of Goss Gilroy Inc.
The evaluation:
- was conducted in accordance with the Treasury Board Policy on Results
- assessed the relevance and effectiveness of the program
- was conducted in 2018
- covered the last 5 fiscal years (2013 to 2014 until 2017 to 2018)
- included documents that date back to 2006 when the program was developed
Results at a glance
- The evidence demonstrates that there is an ongoing need to professionalize the procurement and materiel management communities. The program objective is aligned with federal government priorities and with TBS’s role, although alignment between the program and procurement modernization could be improved by, for example, providing the complex skills and competencies needed for professionalization.
- The program strengthens participants’ procurement and materiel management capacity to some extent; however, the program does not add value to senior-level procurement and materiel management specialists. Issues in program design, delivery and administration present barriers to program participation and the achievement of program outcomes. In addition to issues with the candidate achievement record process, participants are deterred from pursuing or completing the program due to a lack of:
- available courses
- training in complex procurement and materiel management
- courses in strategic thinking
- The functional communities of procurement and materiel management are meeting the business needs of the Government of Canada only to some extent.
- The program does not appear to contribute to the recruitment and retention of functional community members.
Appendix A provides an overview of the certification program and the associated findings of the evaluation.
About the certification program
-
In this section
- Highlights of the certification program
- Background of the certification program
- Structure of the certification program
- Governance: roles and responsibilities
- Champion, the Comptroller General of Canada
- Treasury Board Advisory Committee on Contracting
- Procurement and Materiel Management Advisory Committee
- Certification Program Working Group
- Objectives and design of the certification program
- Expected outcomes
The goal of procurement modernization is to achieve a procurement system that:
- is accessible
- provides value for money
- advances the government’s socio-economic objectives
- simplifies the procurement system for suppliers and client departments
- delivers better results for Canadians
Building a modern procurement process and a professional workforce are key pillars of procurement modernization.
Highlights of the certification program
The program supports Government of Canada employees who are responsible for procurement and materiel management. The program is available to more than 4,000Footnote 1 procurement and materiel management specialists, and thus it has an important role to play in modernizing procurement.
Federal procurement is valued between $19 and $20 billion annually, which represents 14% of Canada’s gross domestic product.Footnote 2
The legislative, regulatory and policy framework of Canada’s federal procurement and materiel management regime includes:
- 15 acts of Parliament
- more than 35 policies
- various trade agreements, statutes and regulations, departmental policies, duties, rules and processes
Procurement modernization and e-procurement were both priorities in Budget 2018 and in the mandate letters for the Minister of Public Services and Procurement Canada (PSPC) and the President of Treasury Board. Overall, the program has had 1826 registered participants and only 108 (6%) graduates.
Background of the certification program
A 2002 discussion paper on “the establishment of professional certification for materiel, supply and real property professionals in the federal government”Footnote 3 concluded that:
- a certification program is “very achievable”
- the work to date to develop a training curriculum is a solid base for establishing the core competencies required in a certification program
- there are a number of viable options for implementing certification, each option depends on the objectives of the government and the financial resources available to support these objectives
Based on these conclusions, the certification program, which was created in response to the Federal Accountability Action Plan, April 2006, needs to “provide accreditation and training for procurement officers.”Footnote 4 The initial goals of the program were to:
- raise the level of professionalism in the procurement and materiel management communities
- recognize that the communities form a knowledge-based profession
The program is subject to the Directive on the Administration of Required Training and the Policy on Learning, Training, and Development.
Structure of the certification program
The program has the following stakeholders:
- TBS’s Procurement, Materiel Management and Real Property Communities Management Office (PMMRP CMO) manages the program and “provides strategic direction and central leadership for the collaborative development and implementation of strategies, programmes and initiatives to support capacity building, community development and the professional recognition of the Federal Government Procurement, Materiel Management and Real Property Communities.”Footnote 5
- PSPC, represented by the Canadian General Standards Board (CGSB), administers the program through a memorandum of understanding (MOU) with TBS. A PSPC-CGSB panel composed of procurement and materiel management experts reviews the functional competencies.
- TBS also consults with the Integrated Services Branch, Professional Training and Development Division of PSPC to determine the equivalency of courses between development programs and the certification program.
- Although the Canada School of Public Service (CSPS) is the official training provider, PSPC is exclusively responsible for providing 3 courses:
- 1 course in Level I procurement
- 2 courses in Level II procurement
- The Public Service Commission of Canada (PSC) provides assessment services and evaluates the core competencies of participants through an MOU between TBS and the Personal Psychology Centre of the PSC.
Governance: roles and responsibilities
Figure 1 - Text version
Figure 1 shows the governance of the certification program as a pyramid with 4 levels on the left and a description of each level on the right. The 4 levels of the pyramid and the accompanying descriptions are:
- Champion, which is described as the Comptroller General of Canada
- Treasury Board Advisory Committee on Contracting, which is described as chaired by the Treasury Board of Canada Secretariat Acquired Services and Assets Sector and composed of executives from the procurement community
- Procurement and Materiel Management Advisory Committee, which is described as chaired by the Treasury Board of Canada Secretariat Communities Management Office and composed of executives from the procurement and materiel management functional communities
- Certification Working Group, which is described as chaired by the Treasury Board of Canada Secretariat Communities Management Office and composed of senior procurement and materiel management officials from various departments and agencies
Champion, the Comptroller General of Canada
The Comptroller General of Canada is responsible for providing functional direction and government-wide assurance for procurement and materiel management. The Comptroller General, through PMMRP CMO, supports the procurement and materiel management communities by:
- maintaining and building the professionalism of the communities through recruitment and development activities
- advocating for the communities and representing their interests at senior-level executive forums
- gathering stakeholder support
Treasury Board Advisory Committee on Contracting
Treasury Board Advisory Committee on Contracting considers matters involving the form and procedures of government contracting and the policy and guidelines on contracting.
Procurement and Materiel Management Advisory Committee
The committee:
- provides a challenge role to the strategies and initiatives brought forward that affect the structure and funding of the communities
- considers recommendations from the PMMRP CMO and working groups
Certification Program Working Group
The working group:
- is used as a platform to discuss:
- ongoing issues with the program
- potential improvements and program modifications
- considers or develops specific initiatives to enhance the program
- brings forward issues for consideration to the Procurement and Materiel Management Advisory Committee
Objectives and design of the certification program
The certification program aims to strengthen and professionalize the procurement and materiel management communities so that:
- they contribute to the effective management of assets and acquired services in the Government of Canada
- they are qualified and able to contribute informed advice on procurement strategies
The program provides both core and function-specific knowledge and skills that lead to better life-cycle management of the federal government’s assets. It targets procurement and materiel management employees at all levels, from entry-level to senior executives.
The program is not mandatory and offers 3 professional designations:
- Certified Federal Specialist in Procurement, Level I, which is certification for junior procurement specialists
- Certified Federal Specialist in Procurement, Level II, which is certification for senior procurement specialists
- Certified Federal Specialist in Materiel Management, Level I, which is certification for specialists in materiel management
The first 2 designations are based on the 2014 Federal Government Procurement Communities Competency Suite and the third is based on the 2014 Federal Government Materiel Management Communities CompetencyFootnote 6 Suite.
Expected outcomes
The expected outcomes of the program, as shown in the logic model in Appendix B, are as follows:
Immediate
- Participants strengthen their procurement and materiel management capacity (knowledge, skills and competencies)
- Participants understand the complex procurement and materiel management environment
- The certification program adds value to the careers of functional community members who participate in the program
- Functional communities are aware of, and senior managers support, the certification program
Intermediate
- Functional community specialists meet the business needs of the Government of Canada
- Departments and agencies attract and retain functional community specialists
Long-term
- Functional communities contribute to the effective and strategic management of assets and acquired services in the Government of Canada
The PMMRP CMO costs for the program between the 2015 to 2016 fiscal year and the 2018 to 2019 fiscal year are shown in Appendix C.
Evaluation methodology and scope
The evaluation assessed the program’s relevance and performance (effectiveness and efficiency) by using multiple lines of evidence in proportion to the program’s risk and materiality. A theory of change was elaborated with the Acquired Services and Assets Sector Communities Management Office to identify:
- the outcomes of the program
- the assumptions or considerations needed for the desired results to occur
The evaluation assessed the extent to which the immediate and intermediate outcomes of the program have been achieved.
Appendix D provides a detailed description of the methodology. The lines of evidence included:
- a document review (including special studies, briefings and a human resources strategy)
- a review of administrative data
- 44 key informant interviews
- an online survey of the procurement and materiel management employees (1032 responses)
- 2 focus groups with 13 clients of procurement and materiel management
The evaluation did not assess the program’s learning components, that is, the appropriateness of the curricula and competencies.
Limitations of the evaluation
The survey population was derived through members of the Procurement and Materiel Management Advisory Committee because it included more than 90% of the communities. However, it did not include small departments. Evaluators mitigated this limitation by reaching out to small organizations, 7 of which participated in the survey.
Relevance
Conclusion: The evidence demonstrates that there is an ongoing need to professionalize the procurement and materiel management communities in the Government of Canada, which is supported by the certification program.
The evaluation found that the program is aligned with federal government priorities and with the role of TBS. However, the evaluation also found that the program’s alignment with procurement modernization could be improved.
Findings: ongoing need for the program
All lines of evidence are consistent in showing that there is a need for the program.
The documentation review indicates that in its role as the employer, TBS aims to make the federal public service a model workplace where skilled workers are trained and motivated to serve Canadians.
Certification or licensure is a common attribute of a profession. It raises the professionalism of the community, recognizes a community as a qualified and knowledge-based profession, and provides value by ensuring that employees have the required foundational skills.
Most interviewees and almost half of survey respondents agreed that certification can promote standardization across the Government of Canada by providing minimum standard criteria (competencies, knowledge and education) for functional communities.
Only 48% of survey respondents think that certification is a valid requirement for the communities. Some respondents think that certification should be voluntary.
The document review shows there is a need to professionalize the community. An objective of modern comptrollership and procurement modernization is to improve the capacity to deliver complex procurement.
The Directive on the Administration of Required Training lends support to the certification program as a valid requirement for the procurement and materiel management communities.
One of the OECD recommendations for effective implementation of public procurement reform is to “develop a procurement workforce with the capacity to continually deliver value for money efficiently and effectively.”
Findings: alignment with priorities, roles and responsibilities
All lines of evidence show that the objectives of the program are aligned with the objectives stated under “Better Government for Canadians” in Budget 2016, which committed to strengthening the public service in order to ensure that it is innovative, agile, collaborative and high-performing. In addition, the role of TBS as enabler involves helping organizations improve management performance and program results.
The documentation shows that capacity building is a component of both procurement modernization and the renewed policy suite for Assets and Acquired Services. However, the document review and most interviewees confirmed that the program is not yet aligned with procurement modernization and the modern approach to comptrollership. The evaluation team verified that the program is based on the 2014 core and functional competencies for procurement and materiel management, which emphasize a transaction-based approach rather than a more tactical and strategic one.Footnote 7
In October 2018, the PMMRP CMO issued the new Government of Canada Procurement Community Competencies, which include a set of 5 technical competencies. The program intends to implement them, pending the results of this evaluation.
While efforts are being made outside the program to advance the modernization of procurement,Footnote 8 some interviewees agreed that the program has not kept pace. Procurement and materiel management are changing with analytics, agile practices, digitization, and new evaluation technologies and tools being used to deliver results. Moving to more strategic thinking would:
- imply a culture change in the way procurement is done
- indicate how the community needs to be supported
Findings: development programs for procurement and materiel management
The evidence shows that the certification program complements development programs established in a few large departments and agencies to recruit and develop entry-level employees in the purchasing and supply (PG)Footnote 9 group. The evaluation found that there is no formal development program in the Government of Canada for the procurement and materiel management communities.
The document review and interviews show that, due to operational requirements, a few departments and agencies that manage complex procurement have created developmental programs to recruit, develop and retain purchasing and supply specialists. These programs develop candidates to the PG-04 level. PMMRP CMO has worked with these departments on components, such as course equivalencies and exams, in an effort to reduce duplication.
The document review and interviews show that the certification program is a component of the Intern Officer Development Program at PSPC and of the Purchase and Supply Development Program at Global Affairs Canada. The program is aligned with the Materiel Acquisition and Support Officer Development Program for the Procurement Group at National Defence and participation in the program is encouraged.
Performance
-
In this section
Management and administration
Conclusion: management and administration
The evidence shows that the program’s structure (organizational structure, roles, leadership and mechanisms) needs improvement. The evidence also shows a need to rethink how to effectively and efficiently assess competencies, which may mean significantly improving the assessment of the candidate achievement records (CARs). The level of investment and leadership has not allowed the program to keep pace with other functional communities in comptrollership.
Findings: A few interviewees stated that the certification program lacks a strong vision and needs to professionalize to support the continued development of the procurement and materiel management communities. Although improvements have occurred, the lack of investment and adherence to program administration requirements (for example, response times to enrollees) has put the integrity of the program at risk. According to TBS key informants, it is difficult to coordinate the program’s stakeholders (PSPC-CGSB, CSPS, PSC and PSPC).
CSPS’s mandate is to provide core and common training. It does not have the authority to provide advanced training. CSPS recently changed the certification program’s curriculum and how the program is delivered, moving from in-class training to online training. As part of these changes, some courses were retired and not replaced (for example, C279, G243, T008), other courses were redesigned, and some courses are being developed. These courses are essential to certification, which means that the program cannot offer a complete curriculum to its learners, and participants cannot complete the required training. This creates frustration and increases the estimated time needed for certification.
PSPC is the sole provider of 3 of the program’s required courses. Interviewees indicated that this presents a challenge since PSPC employees have priority for those courses, which limits access to those courses for other employees.
PSPC-CGSB is the administrator of the program and it:
- maintains the database of candidate files
- performs registrar duties (such as registering students, deciding on course equivalencies, holding exams)
- administers review panels
The document review shows that resourcing issues at PSPC-CGSB have led to significant delays in:
- reviewing CARs
- providing feedback to participants or answering their questions
To accelerate the review of CARs, TBS engaged the PSC in 2018 to help assess core competencies, which helped shorten the CAR reviews.
Some interviewees believe that there are gaps in:
- clearly communicating the requirements of CARs and changes in the program
- providing feedback on results
Most interviewees and 34% of survey respondents enrolled in the certification program identified clear communication of the CARs as the main barrier to the success of the program. The process of reviewing CARs is not considered effective.
Design and delivery
Conclusion: design
The evaluation shows that the design of the certification program is hampering the achievement of its expected outcomes.
Findings: According to the document review, the program was designed to provide procurement and materiel specialists with general knowledge. Documentary evidence revealed that the program was not initially intended to move participants along a formal career path toward certification, given that it was voluntary and did not actively recruit participants via incentives such as promotions. According to key informants, there is an increasing need to formalize the program so that it offers career building and professionalization. Such a redesign would bring the procurement and materiel management communities in line with how other functional communities are professionalizing.
The document review shows that some participants face challenges meeting the experience requirements of the CAR because of the type of procurement work they perform (for example, non-operational procurementFootnote 10). Requirements for CARs have not been aligned to:
- reflect the changes in government procurement (such as new procurement systems, processes or tools)
- consider that procurement is a multidisciplinary function composed of operational and non-operational procurement
Some interviewees who indicated that they were not interested in the program stated that the program is not useful for their work.
The certification program curriculum reflects the competencies found in the 2014 Federal Government Materiel Management Competency Suite and the 2014 Federal Government Procurement Competency Suite (see Table E1 in Appendix E).
According to interviewees and the document review, the 2014 procurement competencies do not meet the needs of the community because they focus on process rather than on strategic thinking. Moreover, adding risk management to the curriculum would be beneficial because risk management reflects changes in the business environment of government procurement and innovation. The 2018 competencies (see Table E2 in Appendix E):
- improved language and usability
- expanded proficiency levels
- added technical competencies that focus on negotiations, project management, risk management, data analytics and business acumen
Conclusion: delivery
All lines of evidence show that the program delivery model is not meeting the needs of TBS, PMMRP CMO, program partners, and stakeholders in terms of training and review of competencies. The way that the program is delivered needs to be significantly strengthened.
Findings: Most interviewees cited the lack of available courses and the inability to choose the language in which the course was delivered as obstacles to completing the program. These obstacles:
- result in delays in program completion
- create frustration
- reduce active participation in the program
For instance, based on course activity in the last 2 years, only 375 active participants have taken a course or registered for an equivalency. Similarly, survey results show that:
- some participants have been enrolled in the certification program since 2006, even though the program is supposed to take only 3 to 4 years to complete
- 78% of the participants who have been enrolled since 2006 are enrolled in Level I
- less than half of the participants have completed the training requirements
- only a few of the participants have completed their CARs
- according to administrative data, between the 2006 to 2007 fiscal year and the 2018 to 2019 fiscal year, 1,826 employees were enrolled in the program (see Figure 2), but only 108 participants were certified
- since 2006, only 5.9% of those enrolled have been certified
In the 2015 to 2016 fiscal year, the highest number of enrolments were reported since the program’s launch. The increase is attributed to the inclusion of questions about the certification program in the Management Accountability Framework and increased communication and outreach efforts by the PMMRP CMO.
Figure 2 - Text version
Figure 2 is a clustered column chart that shows the number of employees enrolled in the materiel management and procurement certification programs between the 2006 to 2007 fiscal year and the 2018 to 2019 fiscal year.
| Fiscal year | Number of employees enrolled in the materiel management certification program | Number of employees enrolled in the procurement certification program |
|---|---|---|
| 2006 to 2007 | 54 | 225 |
| 2007 to 2008 | 21 | 168 |
| 2008 to 2009 | 3 | 97 |
| 2009 to 2010 | 2 | 74 |
| 2010 to 2011 | 3 | 46 |
| 2011 to 2012 | 16 | 81 |
| 2012 to 2013 | 9 | 67 |
| 2013 to 2014 | 3 | 38 |
| 2014 to 2015 | 5 | 60 |
| 2015 to 2016 | 44 | 248 |
| 2016 to 2017 | 62 | 217 |
| 2017 to 2018 | 71 | 236 |
| 2018 to 2019 | 55 | 141 |
Source: administrative data, PSPC-CGSB, 2018 to 2019 Canadian General Standards Statistical Report
Despite the increase in enrolments in recent fiscal years, administrative data show consistently low completion rates, which is one of the program’s most critical issues. As of February 2019, 102 participants had been certified since the start of the program (see Table 1). Approximately 42.5% of participants who have received certification were certified between 2017 and 2019.
Six employees have obtained Level I Certification of Advanced Standing as a Certified Federal Specialist in Procurement as of the program’s launch.
| Type of certification | Number of employees certified |
|---|---|
| Certified federal specialist in procurement: Level I | 90 |
| Certification of advanced standing as a certified federal specialist in procurement: Level I | 6 |
| Certified federal specialist in materiel management: Level I | 9 |
| Certified federal specialist in procurement: Level II | 3 |
| Source: statistics about certified personnel from PSPC-CGSB | |
Immediate outcomes
-
In this section
- Expected outcome: participants strengthen their procurement and materiel management capacity
- Expected outcome: participants understand the complex procurement and materiel management environment
- Expected outcome: the certification program adds value to the careers of functional community members who participate in the program
- Expected outcome: functional communities are aware of, and senior managers support, the certification program
- Awareness
- Procurement community managers
Expected outcome: participants strengthenFootnote 11 their procurement and materiel management capacity
Conclusion: All lines of evidence show that this outcome has been achieved to some extent. Participants have increased their basic knowledge (although the improvement is only somewhat due to the training courses). However, there needs to be a focus on strategic thinking, complex procurement and interpersonal skills.
Findings: The document review shows that despite improvements to the certification program, there are still capacity issues in the procurement community.
Furthermore, the following was noted during the document review:
- a lack of knowledge and experience in the procurement and materiel management communities
- challenges in recruitment and retention of procurement and materiel management personnel
- the lack of a government-wide approach for developing and managing the talent of procurement specialists
The survey and interview results were not consistent with each other. A majority of survey respondents (58%) who are enrolled in the program agree that the program in its current form strengthens their knowledge and skills. However, most interviewees think that only junior employees benefit from the program.
There is evidence that the certification program increases participants’ basic knowledge (such as their knowledge of legal authorities), but the data show that the program does not address how to solve complex problems in procurement and materiel management nor does the program include hands-on training to improve strategic thinking. The curriculum of the program does not cover:
- the strategic advisor role
- the concept of supply chain management
- the management of complex procurement
A few interviewees also said that training is neither well supported nor available. For example, the course “Writing in Clear and Concise Language” has been retiredFootnote 12 even though it provides training in a key skill and its absence is a concern for interviewees. Another example is a course in project management in procurement that was only offered once as a pilot.
Expected outcome: participants understand the complex procurement and materiel management environment
Conclusion: The evidence shows that the program does not equip participants to understand and effectively navigate the complex procurement environment in the Government of Canada.
Findings: According to interviewees, complex procurement and materiel management varies by departmental contract authority. Therefore, in order to gain experience in this area, many specialists have to transfer to a large department. The departments and agencies that have the most complex projects are PSPC, Shared Services Canada and National Defence.
The document review, survey and interview results indicated that the program does not necessarily provide adequate training in complex procurement. Only 47% of survey respondents agreed that they have better skills to take on more complex assignments as a result of training.
Focus group discussions with procurement clients indicated that there is a lack of capacity in the procurement and materiel management communities with respect to soft skills, specifically in navigating the Government of Canada environment. Clients felt that, in today’s environment, key elements to successful procurement are effective communication and client relationships in order to obtain appropriate information and have a deeper understanding of business goals.
Key informants indicated that none of the current courses deal with complex procurement or complex issues and this contention was verified by the evaluators. For example, the “Complex Procurement” course (PSPC 02225) has been under development for some time. This gap was also observed by some CAR reviewers. To help employees broaden their expertise in complex procurement, some large departments and agencies support employees who wish to take university courses and programs.
The document review shows that the Office of the Procurement Ombudsman raised issues about the experience or knowledge of the procurement community in its annual reports. It was mentioned that federal organizations “do not have sufficient procurement staff or have staff that do not have the experience or knowledge needed to tackle the volume and complexity of federal procurement in a way that is fair, open and transparent.”Footnote 13
Expected outcome: the certification program adds value to the careers of functional community members who participate in the program
Conclusion: All lines of evidence show that the program only adds value to the career paths of procurement and materiel management specialists to some degree.
Findings: The document review shows that there is no clear government-wide career path or roadmap for the professional development of procurement and materiel management specialists. The program provides baseline knowledge and is useful largely for entry-level positions. Both the document review and interviews show that a few large departments and agencies are using university programs to train senior-level procurement and materiel management specialists and managers.
Many (49%) survey respondents agreed that the program is recognized by the Government of Canada, although the interviews indicated that not all senior managers are supportive. Among managers, the relevance of the program declines after employees have a few years of experience.
Most key informants indicated the following disincentives to certification, which may be the main reasons why 34% of all survey respondents did not see value in certification:
- Certification does not seem to be a priority for hiring managers; certification appears only as an asset criteria in some job postings.
- Certification is not mandatory for those working in procurement and materiel management and does not result in promotion.
- The program is not comparable to a private industry designation. A private industry designation includes a third-party endorsement (which was initially provided by the PSPC-CGSB), post-secondary requirements, and a maintenance process.
Some interviewees indicated that development programs (for example, the Intern Officer Development Program) within departments and agencies helped to show employees a career path and retain entry-level employees. When asked if certification can be used as a career tool for professional development, most key informants and most (57%) of the survey respondents agreed.
Most survey respondents indicated that they would be more likely to pursue certification if:
- CARs were better managed (52%)
- courses were more readily available (63%)
- courses were more often available in their regions (47%)
- the program was a requirement for promotion (60%)
Expected outcome: functional communities are aware of, and senior managers support, the certification program
Conclusion: Functional communities are aware of the program to some extent, and senior manager support varies across the Government of Canada.
Awareness
Though all survey respondents indicated that they are aware of the certification program, the majority (62%) agreed that the demands and pressures on their current job limit their ability to participate in it. This was confirmed by most key informants who also believe that the communities are aware of the program. However, it was recognized that more awareness is needed in the regions and in small departments and agencies.
Procurement community managers
There were mixed views on management’s support of the program. Some key informants believe managers have been supportive by:
- promoting the program
- providing the resources needed for completing the program
- embedding the program in employee performance management agreements
Others believe there is no buy-in from middle management and that there is support in principle but not in practice.
On the other hand, a majority of survey respondents (58%), many of whom were already enrolled in the program, agreed that their management supports their pursuit and completion of certification.
The document review showed that procurement and materiel management managers are short-staffed and feeling overwhelmed, making it difficult for them to support a program that is not mandatory.
Intermediate outcomes
Expected outcome: functional community specialists meet the business needs of the Government of Canada
Conclusion: All lines of evidence show that procurement and materiel management specialists only meet the business needs of the Government of Canada to some extent.
Findings: The document review shows that there is a lack of capacity in procurement to meet the business needs of the government. In the Annual Report 2016 to 2017 from the Office of the Procurement Ombudsman, the lack of capacity among procurement staff in the Government of Canada was cited as:
- a reason for delays during various stages of the procurement process
- the possible cause of an increasing reliance on non-specialists to undertake some procurements
Moreover, the report stated that “the lack of capacity results in experienced and knowledgeable procurement specialists being highly sought after, with federal organizations routinely poaching staff from one-another.”
Even though half (50%) of the survey respondents who are enrolled in the program agreed that they have a better understanding of the business needs of the Government of Canada, most key informants believe that procurement and materiel management specialists do not meet the business needs of the Government of Canada because not all of them are equipped with the skills and abilities needed to provide strategic advice.
Most interviewees said that the focus of the procurement and materiel management function is changing, and there is a need to move from a transactional, risk-averse approach to an agileFootnote 14 process (outcome-based procurement and the ability to negotiate with complex industries). But procurement and materiel management specialists are still trained to use procedures and rules. Interviewees stated that procurement specialists are like a fish out of the water in the new federal environment. In some cases, they felt that procurement and materiel management specialists do not understand how their daily tasks tie into the strategy or priorities of their department and the government.
In an effort to meet government business needs, TBS:
- has linked investment planning, project management, materiel management and real property to procurement
- has updated the policy suite in procurement
- is considering establishing procurement leadership within organizations (piloting the chief procurement officer role in departments to support deputy heads)
PMMRP CMO is also increasing collaboration with departments and agencies on how to address the need for professionalization.
Expected outcome: departments and agencies attract and retain functional community specialists
Conclusion: The program does not appear to contribute to attracting or retaining functional community specialists.
Findings: All evidence shows that most departments and agencies face challenges attracting employees to the procurement and materiel management profession. Indeed, some members of those communities are leaving for other functions. A key concern is the retention of experienced employees above the PG‑04 level.
Only a third (34%) of survey respondents believe that having a certification program helps retain specialists. The most common challenges mentioned by interviewees to attracting and retaining procurement specialists are:
- a lack of recognition as a profession, particularly among post-secondary graduates and the private sector procurement community
- salary differences with the private sector
- work requirements, such as language and security
- a poorly defined government-wide career path
Most departments and agencies attract specialists by offering desirable work environments, such as flexible work arrangements, work-life balance, training and mentoring. Some use co-op programs and the Federal Student Work Experience Program to hire students from post-secondary supply chain management programs. A few large departments and agencies have development programs for entry-level positions (such as the Intern Officer Development Program) or have taken on Intern Officer Development Program candidates from PSPC.
Most key informants commented on the lack of a government-wide human resources strategy to attract and retain employees in procurement. Departments and agencies have had a more reactive focus and have run staffing processes when they needed to fill vacancies rather than planning to ensure that they have the capacity that they need. More recently, PMMRP CMO developed a government-wide human resources strategy for procurement (a shorter version of which was published on GCpedia) and ran collective competitions for PG‑02, PG‑04, PG‑05 and PG‑06 positions.
Alternatives to the current program design
Conclusion: The interviews, documentation and literature review indicated other ways to deliver the certification program, although a thorough assessment of these was outside the scope of the evaluation.
One option is adopting a graduated approach to development that includes certifications at different levels (for example, progression to PG‑06). This would ensure that employees have the required knowledge by level and would allow participants to be assessed more uniformly (generalist to specialist to strategic specialist). The graduated levels could be described as follows:
- Levels 1 and 2: general procurement certification to develop a baseline in technical knowledge and competencies. Certification at this level would be equivalent to PG‑01 to PG‑03.
- Level 3: a generalist may obtain a specialist certification by completing a variety of advanced courses and training. Certification at this level would be equivalent to PG‑04 to PG‑05. Candidates who wish to be certified at Level 3 may need to pass an exam.
- Level 4: a specialist procurement officer may become a strategic analyst in procurement (PG‑06) by demonstrating a variety of skills, competencies and knowledge in areas such as data analytics and strategic advice.
The graduated approach by levels is similar what is in place in Australia and the United States (see Appendix F). In the United States, certification is mandatory and there are 3 different levels of certification that cover project managers for advanced procurement, professional procurement and contract management officers.
Another option is transferring administration of the certification program to TBS from PSPC-CGSB. Though a few interviewees suggested that PSPC should be the administrator because of its role in procurement operations and the high number of PG positions in the department, others suggested that TBS should administer the program. TBS was proposed because:
- it is the policy centre for assets and acquired services
- the Comptroller General is the champion for the procurement community
Yet another option noted by a few interviewees is using a private service provider for accreditation purposes. The literature review shows there are some accreditation service providers in Canada that have different focuses. The possible benefits of using a private service provider are international recognition, lowered costs and clear requirements.
Recommendations
- It is recommended that PMMRP CMO strengthen program management design and establish alternative service providers for training.
- To reduce barriers and increase the program’s relevance for learners, PMMRP CMO should:
- consider a graduated approach to certification that targets the emerging and future needs of the procurement and materiel management system and includes the critical component of complex procurement
- determine a more effective and efficient way, than the CAR process, of assessing competencies based on jurisdictional research
- In line with how other functional communities are professionalizing, PMMRP CMO should develop a career-path framework for procurement and materiel management specialists and explore the possibility of developing a common, government-wide development program that includes certification.
Appendix A: journey to become a certified federal specialist in procurement and materiel management and related evaluation findings
| Step 1: enrolment in the certification program | |
|---|---|
| Information about enrolment | Prerequisites for the certification programs depending on level and focus:
A consent form is required for the assessment of core competencies. There are no enrolment costs for the employee. The estimated time needed to successfully complete the program is 3 to 4 years. |
| Evaluation findings about enrolment |
Comments from survey respondents
|
| Step 2: training provided as part of the certification program | |
| Information about the training provided |
Procurement
Materiel management
|
| Evaluation findings about the training provided |
Comments from survey respondents
|
| Step 3: the candidate achievement record process | |
| Information about how to complete a candidate achievement record |
|
| Evaluation findings about the candidate achievement record process |
Comments from survey respondents
|
| Step 4: exam | |
| Information about the exam | Before writing the exam
Procurement Level I: multiple-choice exam composed of 92 questions (answer 70 questions correctly in order to pass) If you fail:
Level II: case-study exam that does not have a pass mark (candidates are scored as meets expectations or does not meet expectations) Materiel management Level I: multiple-choice exam (structure similar to the exam for Procurement Level I) administered by the Public Service Commission of Canada |
| Evaluation findings about the exam | Of those survey respondents who have completed all course requirements, most have tried to write the Level 1 exam. Of those who have not written the exam, the major reasons for not having done so are:
|
| Step 5: certification | |
| Information about certification | Once a candidate has passed their exam, they can receive their certification by submitting an application form. Certifications Procurement: Certified Federal Specialist in Procurement (CFSP), Level I and Certified Federal Specialist in Procurement (CFSP), Level II Materiel management: Certified Federal Specialist in Materiel Management (CFSMM), Level I. |
| Evaluation findings about certification | Facts about the certification program
Comments from survey respondents
|
Appendix B: logic model for the certification program
Figure 3 - Text version
The logic model shows, from the bottom to the top, the activities, outputs, reach, immediate outcomes (capabilities, opportunities), intermediate outcomes (behaviour) and long-term outcome (benefit) for the certification program:
Activities
- Administration and maintenance of the certification program
- Monitoring and reporting
- Training and certification
- Outreach and engagement
- Development of community competencies and research
Outputs
- Memorandums of understanding, working groups and governance structure
- Business tools, reports and database
- Program curriculum, certification, exams and candidate achievement records
- Outreach tools and GCpedia
- Functional community competencies
Reach
- Certification program participants
- Procurement and materiel management communities (analysts, managers and senior managers)
Immediate outcomes (capabilities, opportunities)
- Participants strengthen their procurement and materiel management capacity (knowledge, skills and competencies)
- Participants understand the complex procurement and materiel management environment
- The certification program adds value to the careers of functional community members who participate in the program
- Functional communities are aware of, and senior managers support, the certification program
Intermediate outcomes (behaviour)
- Functional community specialists meet the business needs of the Government of Canada, which means that departments and agencies have informed advice on procurement strategies and options
- Departments and agencies attract and retain functional community specialists
Long-term outcome (benefit)
- Functional communities, including the procurement and material management functional communities, contribute to the effective and strategic management of assets and acquired services within the Government of Canada
Appendix C: costs of the certification program and key assumptions for costing
| Source of cost | 2015 to 2016 | 2016 to 2017 | 2017 to 2018 | 2018 to 2019 |
|---|---|---|---|---|
| Program administration (Canadian General Standards Board) | $296,596 | $295,461 | $241,669 | $314,624 |
| Management officetable C1 note * | $75,000 | $75,000 | $75,000 | $75,000 |
| Core competencies: assessments and exam development (Public Services and Procurement Canada) |
$55,435 | $84,562 | $19,504 | $52,272 |
| Total | $430,031 | $458,023 | $339,173 | $442,896 |
Table 1 Notes
|
||||
| Type of cost | Key assumptions |
|---|---|
| Administration |
|
| Opportunity |
|
| Enrolment |
|
| Based on the assumptions, the average cost per participant in the certification program was estimated at $11,315. | |
Appendix D: evaluation methodology
The evaluation was guided by an approved evaluation framework, which was a detailed plan of the evaluation activities, questions and indicators.
Evaluation questions
Relevance
- Is there an ongoing need for certification in procurement and materiel management? What are the needs of the functional communities and the Government of Canada? Is certification needed for the community?
- To what extent does the certification program complement programs in other departments and agencies? Is there overlap or duplication?
Performance
- How appropriate is the program’s governance structure?
- To what extent does the program’s design and delivery enhance enrolment and certification?
- To what extent have the certification program outcomes been achieved?
- What factors have contributed to the achievement of the expected outcomes?
- What have been the barriers to achieving outcomes?
Alternatives
- What best practices and design features can be learned from the certification and professional development programs?
Methodology
Consistent with best practices, the evaluation of the program included multiple lines of evidence to ensure that reliable and sufficient information is produced from both quantitative and qualitative analysis. The methods, summarized below, are:
- a document review
- an analysis of administrative data
- interviews
- an online survey of targeted procurement and materiel management communities
- 2 focus groups with procurement clients
Document review
The document review provided evidence to supplement and expand on the analysis of the administrative data and the survey. The documents included:
- memoranda of understanding (MOUs) and agreements between Public Services and Procurement Canada’s Canadian General Standards Board (PSPC-CGSB) and the Public Service Commission of Canada (PSC)
- discussion papers
- briefings and presentations to various audiences
- a profile of the program
- a summary of procurement in other jurisdictions
- a comparison of different departmental development programs for procurement
- terms of reference for committees and review panels
- annual reports and information from the Office of the Procurement Ombudsman
- applicable policies and directives
- minutes from governance bodies
- information about the certification program published on GCpedia and Canada.ca
- survey results from the Public Service Employee Survey about the procurement and materiel management communities
Analysis of administrative data
The analysis of program administrative data that is maintained by the PSPC-CGSB includes the analysis of the data collected for:
- regular reports prepared for the Treasury Board of Canada Secretariat (TBS)
- the day-to-day management of the program
Key informant interviews
There were 42 interviewees in 33 in-depth sessions with key informants. The interviews were conducted using open-ended questions in 14 departments and agencies, and a session was held with the ombudsman procurement officer. Key informant interviews are a qualitative method used in evaluation to address most evaluation issues and questions. The interviews gather views and factual information from key informants selected from within the federal government and the core public administration.
Interviews were conducted with representatives from:
- the procurement certification program’s management team (5 interviews)
- classification policy at the Office of the Chief Human Resources Officer (2 interviews)
- program administration at PSPC (5 interviews)
- curriculum development at the Canada School of Public Service (2 interviews)
- the PSC (6 interviews)
- National Defence (2 interviews)
- the Procurement and Materiel Management Advisory Committee (2 interviews)
- the departments and agencies that have the largest number of procurement practitioners (7 interviews)
- the Certification Program Working Group (3 interviews)
- the departments and agencies that have development programs for procurement (7 interviews)
- other functional communities in the Government of Canada (2 interviews)
- the Office of the Procurement Ombudsman (1 interview)
The people who were invited to interviews were given the opportunity to organize group interviews. Each interview was weighted equally regardless of the number of interviewees at the interview.
Online survey of the procurement and materiel management communities
Evidence from the procurement and materiel management communities was gathered via an online survey. An attempt was made to contact all Government of Canada employees in the procurement and materiel management communities. Each department and agency was sent a letter from the Office of the Comptroller General and asked to provide a list of employees in the purchasing and supply (PG) occupational group who would be eligible to participate in the certification program. These lists were then provided to the evaluators by the certification program at TBS. Contact information included each participant’s name, email and department. All files were assessed for duplication in order to create a unique set of individuals for sampling.
In total, 3,656 participants were contacted by email to participate in the survey. Of the individuals with valid contact information (3371), 1032 completed the population survey producing a response rate of 31%. The margin of error for this proportion is plus or minus 2.5% at a confidence level of 95%.
For the results of the survey, the following scale was used:
- Few: less than 20%
- Some: 20% to 39%
- Many: 40% to 49%
- Half: 50%
- Most: 51% or more
Focus groups
Two focus groups were held with clients of procurement services. A total of 13 people representing 8 departments participated in both focus groups. The focus group sessions focused on:
- issues that arise when dealing with procurement
- suggestions for improving procurement
- the skills required for procurement
- improvement of the certification program
- service standards
Appendix E: 2014 and 2018 competencies
2014 core and functional competencies for procurement and materiel management
Core competencies for procurement and materiel management
- Values and ethics
- Communication
- Relationship building
- Thinking and judgment
- Adaptability
Functional competencies for:
- Procurement:
- Assessment and planning
- Acquisition
- Managing contract and contract close-out
- Materiel management:
- Managing materiel
- Disposal
Source: 2014 Competencies for Procurement and Materiel Management
2018 core, functional and technical competencies for the procurement community
Core competencies according to the Directive on Performance Management
- Demonstrating integrity and respect
- Thinking things through
- Working effectively with others
- Showing initiative and being action oriented
Procurement community competencies:
- Functional competencies for a particular employee group
- Assessment and planning
- Acquisition
- Managing contracts and close-out
- Technical competencies for a particular job
- Negotiations
- Project management
- Risk management
- Data analytics
- Business acumen
Source: 2018 Government of Canada, Procurement Community Competencies
Appendix F: governmental procurement and materiel management training programs in other countries
| Country | Main characteristics | Best practices |
|---|---|---|
| United States | The United States has a Chief Acquisitions Officer (CAO).
To modernize procurement, the United Stated Digital Service (USDS) and the Office of Federal Procurement Policy (OFPP) developed Digital IT Acquisition Professional Training (DITAP), which is an immersive development program. In 2018, the Office of Management and Budget established the Federal Acquisition Certification in Contracting Core-Plus Specialization in Digital Services (FAC-C-DS). |
|
| United Kingdom | The United Kingdom has a Government Chief Commercial Officer.
|
|
| Australia |
|
The onboarding approach:
|
| New Zealand | The government of New Zealand offers:
The graduate program is a professional skills development program that lasts 2 years. It provides mentoring, a buddy program, networking events and training. The program includes 3 to 4 placements in select government agencies. |
n/a |
Appendix G: Management response and action plan
-
In this section
The Communities Management Office (CMO) in the Acquired Services and Assets Sector, Office of the Comptroller General, Treasury Board of Canada Secretariat (TBS), has reviewed the evaluation and agrees with the recommendations. Proposed actions to address the recommendations of the report are outlined in the tables below.
Recommendation 1
It is recommended that PMMRP CMO strengthen program management design and establish alternative service providers for training.
Management response
TBS agrees that program management design needs to be strengthened and that alternative service providers should be examined and established for program training.
| Proposed action for recommendation 1 | Start date | Target completion date | Office of primary interest |
|---|---|---|---|
|
CMO | ||
|
CMO | ||
|
CMO with CSPS as co-lead | ||
|
CMO |
Recommendation 2
To reduce barriers and increase the program’s relevance for learners, PMMRP CMO should:
- consider a graduated approach to certification that targets the emerging and future needs of the procurement and materiel management system and includes the critical component of complex procurement
- determine a more effective and efficient way than the candidate achievement record process of assessing competencies, based on jurisdictional research
Management response to recommendation 2a
TBS agrees with the recommendation to consider a graduated approach to certification that addresses emerging and future needs within the procurement community, including the critical component of complex procurement.
| Proposed action for recommendation 2a | Start date | Target completion date | Office of primary interest |
|---|---|---|---|
|
CMO | ||
|
January 6, 2020 | March 31, 2021 | CMO |
|
CMO | ||
|
CMO |
Management response to recommendation 2b
TBS agrees with the recommendation to determine a more effective and efficient method of assessing competencies, based on jurisdictional research.
| Proposed action for recommendation 2b | Start date | Target completion date | Office of primary interest |
|---|---|---|---|
|
CMO | ||
|
CMO |
Recommendation 3
In line with how other functional communities are professionalizing, PMMRP CMO should develop a career-path framework for procurement and materiel management specialists and explore the possibility of developing a common, government-wide development program that includes certification.
Management response
TBS agrees that it should:
- develop a procurement and materiel management career-path framework
- research the feasibility of a common, government-wide development program that includes certification
| Proposed action for recommendation 3 | Start date | Target completion date | Office of primary interest |
|---|---|---|---|
|
CMO | ||
|
CMO | ||
|
CMO | ||
|
CMO |
Page details
- Date modified: