2013-2014 Report on plans and priorities: Methodology for the report on plans and priorities targets

Prepared: March 19, 2012

As mandated by the Treasury Board Secretariat (as part of the Management Accountability Framework (MAF) and Management Resources and Results Structure (MRRS) policy), government departments are required to ensure that performance objectives are related to public purpose and overall government priorities and further be associated with actions to achieve results, taking into consideration both past and future realities. Performance information should facilitate comparisons with other benchmarks and industry or government norms and standards and data sources, frequency of data collection and performance targets should then be articulated for all performance indicators. This information must be clear, relevant, credible and balanced.

CSC has developed a Performance Measurement Framework (PMF) associated with the Program Activity Architecture (PAA) that includes internal benchmarking and forecasting techniques in order to meet these requirements. Internal benchmarks are derived from 10 years of historical performance information. The methodology identifies the median or middle value of the data and sets 3 ranges of performance around this value, with 50% of the observable data associated within the middle band (or 'usual' performance zone). Data outside this band are considered either higher or lower than usual. The performance bands are color-coded for ease of interpretation, as follows:

Quartile 4 (Red): Poor Performance Zone

Quartile 3 (Yellow/Red Hatch): Usual Performance Zone*

Quartile 2 (Yellow): Usual Performance Zone

Quartile 1 (Green): Best Performance Zone

* Note: The middle performance zone incorporates quartiles 2 and 3 (or 50% of the data). While both quartiles fall within the 'usual' performance range, they are separated to delineate proximity to the poor and best performance ranges. In effect, quartile 3 serves as a sort of 'caution' zone.

The preferred method of benchmarking and the development of performance zones uses the median or middle value, at which 50% of the data is above the value and the other 50% is below. There are several advantages to using the median: it reduces the impact of outlying data and skewed data, especially when there are only a few data points; it is often a better indicator of where the true "middle" performance lies than the mean, although is still susceptible to extreme outliers. The median value will be the CSC benchmark with three performance zones developed around it:

• Usual performance band - covers the middle 50% of observed data (25^th up to the75^th percentile). Data outside this band are considered either higher or lower than usual. For ease of interpretation, usual performance is denoted by the colours yellow (usual) and hatched yellow (usual-caution);
• Better performance than usual - will either represent the lowest or highest 25% of the data. This performance zone is green;
• Worse performance than usual - will either represent the lowest or highest 25% of the data. The performance zone is red.

Whether performance outside of the usual band will be considered worse or better depends on the indicator being examined.

Example 1. Percentage of Initial Placements Completed on Time (figure 1.)

Example 2. Rate of Violent Institutional Incidents (figure 2.)

In the case of the percentage of initial assessments completed on time, the upper-most band indicates higher/better performance than usual (i.e., more assessments completed); whereas in the case of rate of violent institutional incidents, the lower band indicates better than usual performance (i.e., fewer institutional incidents).

It is important to note that the upper and lower performance zones are based on annual national rates and percentages. Regional and site results may actually fall outside the performance bands at the annual national levels (because the upper and lower bands have been truncated to coincide with the highest and lowest annual national rate or percentage for ease of presentation).

Several forecasting techniques were examined in order to estimate values of indicators in coming years. Of all of the methods examined, the most reliable was chosen for this exercise and is used with all of the indicators. The estimates for the coming year’s fiscal end (i.e., the end of FY 12-13), are averages of the past 18 months performance (i.e., the period starts at April 1, 2010 and ends September 30, 2011 for each of the indicators). Using this technique assumes the coming fiscal year will be similar to the past 18 months.

Two types of data have been used for the RPP:

1. Percentage – the number of individuals in a population with a given characteristic relative to the total number of individuals in a population multiplied by 100. This value can range from 0% to 100%

OR

2. Rate per 100 offender person years (OPYs) – the total number of events in a given period of time divided by the total time at risk for a given population (OPY). The rate is multiplied by 100 to establish context in relation of offender populations. In this document, rate always refers to rate per 100 OPY ^{Footnote 1}.

OR

This indicator adjusts for time-at-risk which enables more meaningful comparisons of performance results across variable periods of time. Events are expressed over the total number of years that offenders are ‘at-risk’ for the event during any given period of time.

For example an annual rate of non-natural deaths in custody of 0.35 means that if 100 offenders were followed for the period of one year we would expect 0.35 non-natural deaths. We can also multiply this rate by 10 if lower rates are present. The same example could then be expressed as "we would expect 3.5 non-natural deaths in custody, if 1000 offenders were followed for a period of one year."