Annex F : Military Justice System Performance Monitoring Framework

Yvon Dandurand

School of Criminology and Criminal Justice, UFV& & International Centre for Criminal Law Reform and Criminal Justice Policy

In co-operation with the Directorate of Military Justice Operations

Report prepared for the Military Justice Division 

Office of the Judge Advocate General 

National Defence Headquarters

August 6, 2019

Forward

The Directorate of Military Justice Operations, within the Military Justice Division of the Office of the Judge Advocate General, has been generous in assisting me in the preparation of this Report. In particular, Major Greg Koenderman has helped guide my understanding of Canada’s Military Justice System and has provided helpful feedback on draft versions of this Report.

Introduction

The goal of this project is to develop a performance monitoring framework (PMF) capable of delivering ongoing, objective, meaningful, and relevant data on the performance of the military justice system (MJS). This project is undertaken at a time when a parallel project is developing a Justice Administration and Information Management System (JAIMS) for all key military justice activities of the Office of the Judge Advocate General and the Canadian Armed Forces. As the performance monitoring framework is expected to rely heavily on the administrative data that will soon be available out of the information management system, it is important for the two projects to proceed in close consultation. This will help ensure that the chosen performance indicators are aligned, to the extent possible, with the data generated by JAIMS.

The purpose of this report is to identify, as a basis for internal consultations, the main elements of a performance monitoring framework for the MJS. The proposed performance framework includes more than two dozen indicators grouped under twenty or so main dimensions. A first group of indicators relates to the workload and activities of the MJS, a second group of indicators relates to the outputs of the MJS, while a third group relates to its immediate and longer-term outcomes. The use of military justice indicators relies on a process through which information about the MJS is collected, packaged and communicated so as to serve as a basis for learning, experimenting and decision-making within that system.

Following approval of the present proposal, the indicators and related metrics will be assessed and confirmed through testing of the framework based on available data.

Background

In the public sector, performance measurement systems serve at least three main purposes: (1) improve performance management by providing timely feedback to managers on the activities, outputs and outcomes for which they are responsible; (2) provide a basis for an internal or external accountability structure for an organization; and, (3) support greater public transparency. The development of a PMF must keep these three objectives in mind, particularly in a context where managers are held accountable for delivering results. A strong PMF can generate a virtuous feedback loop to support organizational change and reforms in the MJS. A PMF is a useful tool to monitor performance, draw attention to issues, establish benchmarks, monitor progress, and evaluate the impact of changes and reforms. Together with other monitoring and evaluation mechanisms, a PMF is essential to providing feedback to policy makers and managers and, when made public, to contribute to greater transparency and public accountability.

In the case of the MJS, Canadian Armed Forces’ (CAF) highest-ranking legal officer, the Judge Advocate General (JAG), is responsible for the superintendence of the administration of military justice in the CAF. Based on a review of the MJS, we can infer that the system’s effectiveness, efficiency, and legitimacy are organizational goals and express the essential performance dimensions to be captured by a PMF for the MJS.

Defining, and eventually articulating, these performance dimensions will ensure that appropriate metrics are identified to monitor the outputs and outcomes of the MJS in relation to these dimensions. For the purpose of the present exercise, the definitions found in the Oxford English Dictionary, pending further elaboration, can provide a starting point:

  1. Efficient: achieving maximum productivity with minimum wasted effort or expense;
  2. Effective: successful in producing a desired or intended result; and
  3. Legitimate: conforming to the law or to rules or able to be defended with logic or justification.Footnote 1

A PMF that can contribute to enhancing the effectiveness, efficiency, and legitimacy of the MJS must operate at two levels: (1) the level of the MJS as whole to measure the extent to which it is efficient, effective, and legitimate (performance of the MJS as a whole); and, (2) the level of MJS’ various components to measure their individual performance as well as their contribution to the overall performance of the MJS.

Back to Top

Main Characteristics of the PMF

Performance needs to be defined in a context-specific manner. For that purpose, we must at the outset articulate the most relevant elements of the MJS’ performance, in relation to each of the key performance dimensions identified above (efficiency, effectiveness and legitimacy) as well as the specificity and legal basis of the military justice process.

It is possible to distinguish between major categories of justice indicators: input or activity indicators, output indicators, and outcome or strategic indicators. All three types of indicators are relevant to performance management.

  • Input indicators measure workload, activities, and the resources at the disposal of the system.
  • Output indicators measure the outputs produced by the system and its various components and, when compared to input indicators, targets, or benchmarks, provide a measure of efficiency. When compared to normative or legal standards, outputs can also provide a measure of legitimacy.
  • Outcome (or strategic) indicators measure performance against the broad objectives of the justice system (e.g., maintaining discipline, public safety, perceived legitimacy of the system as a whole, public confidence in the military justice system, fairness, etc.).

Comparisons between input and output indicators provide the basis for monitoring the efficiency of the justice system. Measures of outputs defined in relation to law or other normative standards offer a basis for measuring the legitimacy of the justice system. Measures of outcomes focus on the effectiveness of the system (the results it achieves) in relation to the overall goals and objectives of the system.

It is neither necessary nor feasible for a PMF to measure everything. Although it may be tempting to develop an exhaustive list of indicators that cover all aspects and processes of the MJS, what is likely to be most useful is a comprehensive framework covering the relevant performance dimensions based on a relatively small number of key performance indicators (KPIs) and associated metrics. A manageable and sustainable PMF should limit itself to relatively few simple and unequivocal KPIs.

Choices have to be made about the KPIs that will be retained in the PMF. These choices are dictated by aspects of performance that need to be covered, the purpose to be served by each indicator, data availability, and the level of efforts and resources required to measure these indicators. The fact that the PMF is being developed concurrently with the development of JAIMS presents a unique opportunity to ensure that the information management system collects and reports data in a manner and format consistent with the PMF.

Several factors guide the development and implementation of a PMF, including the need for clarity about the objectives and scope of the framework, a preference for actionable and dynamic indicators, and the need for stable yet flexible measures. The framework should preferably rely on dynamic indicators, that is, indicators with a demonstrated capacity to capture and reveal some of the subtler changes in outputs and outcomes.

Measuring change over time is a crucial part of the exercise. Indicators are most revealing when the same measure is tracked over time, or at least against some baseline data. Successive or periodic data collection exercises make it possible to identify trends and to observe changes in various aspects of the justice system. However, the MJS is itself constantly changing, potentially making some performance indicators obsolete. The indicators must therefore be capable of adapting to changing circumstances while remaining stable enough to monitor change over time and identify trends. As with any performance monitoring framework, it is expected that adjustments and modifications may be required over time, as the framework is being applied and as the MJS evolves or priorities change.

Finally, if necessary, a PMF and its various indicators can be implemented incrementally as the organization’s capacity to produce the necessary metrics increases. It sometimes makes sense to start with a few important indicators and then add to them over time when necessary.

At this point the proposed PMF does not anticipate the use of composite indices or indexes comprised of several indicators. The use of such indexes can be controversial because it can be difficult to come to an agreement on the selection of the component indicators and their relative weights within the index.Footnote 2

Back to Top

Sources of Data

The proposed PMF relies on only two sources of quantitative data: administrative data and, to a lesser extent, survey data. The required administrative data are expected to be derived principally from JAIMS, as well as data available to the Directorate of Military Justice Operations.

Surveys can also be an important source of information on the performance of a system. And while surveys are sometimes thought to provide information that is less objective and thus less reliable than data generated through other means, much depends on the questions asked and also on how and when they are asked. There are obviously questions about the performance of the military justice system that can only be answered by people who, in different capacities, have had a direct experience of it.

Back to Top

Main Dimension of the Performance Monitoring Framework

The primary focus of the proposed PMF is on outcomes. However, to put these outcomes into context and to eventually be able to relate fluctuations in outcomes with variations in inputs, activities, or outputs, the proposed framework also includes measures of the latter.

System’s Inputs and Activities

  • Volume of cases
  • Frequency of solving crime
  • Frequency of military prosecutions
  • Frequency of civil prosecutions
  • Frequency of summary trials (ST)
  • DMP Preferral Rate
  • Frequency of courts martial (CM)
  • Cost of CMs

System’s Outputs

  • Timeliness of process
  • Pre-trial detention
  • Early resolution of cases
  • Outcomes of proceedings
  • Legal assistance provided to accused
  • Summary Trial Reviews
  • Number of judicial reviews (Federal Court)
  • Number of appeals to CM Appeal Court of Canada (CMAC)

System’s Outcomes

  • Outcomes of ST Reviews
  • Outcomes of Federal Court judicial reviews
  • Outcomes of appeals to CMAC
  • Access to justice for accused members
  • Access to justice for victims of service offences
  • Confidence in the MJS among victims
  • Confidence in the MJS among Commanding Officers
  • Confidence in the MJS expressed by CAF members
  • Perceived fairness of the MJS
  • MJS contribution to Efficiency in the CAF
  • MJS contribution to Discipline in the CAF
  • MJS contribution to Morale in the CAF

Back to Top

The proposed Performance Monitoring Framework

The Military Justice Performance Monitoring Framework

DIMENSION (Source) INDICATORSFootnote 3 DESIRED TREND DIRECTION RATIONALE
MJS Inputs and Activities
1. Volume of Cases
(JAIMS)
a. Number of alleged offences reported in yearFootnote 4 Not Applicable (N/A) To put outcomes into context and to be able to relate fluctuations in outcomes with variations in inputs and activities, the PMF includes measures of inputs. The volume of cases entering the MJS is relevant to the understanding and interpretation of certain other indicators in the PMF.
b. Annual change in the number of alleged offences reported in year as compared to previous year expressed as a percentage 
(by type of offencesFootnote 5 )
N/A
2. Frequency of “solving” crime (JAIMS)

a. Clearance Rate (Total number of cases in which a charge is laid divided by total number of offences reported in a year)Footnote 6 

Increasing This indicator will provide insight into the effectiveness of the investigation process and its ability to identify the person responsible for a service offence. The clearance rate will also be relevant to the system’s perceived legitimacy.
3. Frequency of military prosecutions (JAIMS) a. Number of prosecutions initiated in the MJS (defined as a member being charged with one or more offences) in year N/A To put outcomes into context and to be able to relate fluctuations in outcomes with variations in inputs and activities, the PMF includes measures of inputs. The frequency of military prosecutions is relevant to the understanding and interpretation of certain other indicators in the PMF.
b. Number of prosecutions completed in year N/A
c. Annual change in the number of prosecutions initiated in year as compared to previous year, expressed as a percentage N/A
d. Annual change in the number of prosecutions completed in year as compared to previous year, expressed as a percentage N/A
e. Proportion of service tribunals that proceed as summary trials (vice court martial) Greater than 90%Footnote 7 
f. Percent of accused electing to be tried by court martial Between 15% and 25%Footnote 8 Where accused members have an option between types of service tribunal, their choice could be relevant to the understanding of factors like perceived fairness, timeliness, access to justice, and others.
4. Frequency of civil prosecutionsFootnote 9  (JAIMS) a. Number of prosecutions initiated by military authorities in the civilian justice system
(by type of offences)
Decreasing The decision to pursue a prosecution in the civilian system rather than the MJS is relevant to the legitimacy (lawfulness, fairness, etc., including fairness to victims) and could also be related to the issue of efficiency.
b. Annual change in the number of prosecutions pursued by military authorities in the civilian justice system in year as compared to previous year, expressed as a percentage
(by type of offences)
Decreasing
5. Frequency of summary trials (JAIMS) a. Number of summary trials (ST) completed in year
(By type of offences and by type of summary trial: Delegated Officer, CO, Superior Commander)
N/A To put outcomes into context and to be able to relate fluctuations in outcomes with variations in inputs and activities, the PMF includes measures of inputs. The frequency of summary trials is relevant to understand the volume of work moving through the ST system.
b. Annual change in the number of ST completed in year as compared to previous year, expressed as a percentage
(By type of offences and by type of summary trial)
N/A
c. Number of cases in which CO or Superior Commander decides not to proceed with one or more charges(s) laid Decreasing A decision not to proceed with one or more charges is relevant to the effectiveness of the investigative and charge-laying process and the legitimacy of the ST system (lawfulness, fairness, etc., including fairness to victims) and could also be related to the issue of efficiency.
d. Percentage of cases in which CO or Superior Commander decides not to proceed with one or more charges(s) laid Decreasing
6. DMP Preferral Rate
(JAIMS)
a. Number of DMP Preferrals (files referred to DMP where charges are preferred)
(By type of investigation: NIS, MP, Unit)
N/A This indicator provides insight into the effectiveness, efficiency, and legitimacy of the investigative, charge laying, and referral process. If the referral package provided to DMP supports a reasonable prospect of conviction (and it is in the public interest), then the DMP will prefer charge(s) against the accused. The decision not to prefer any charges could suggest a deficiency in the MJS up to the point of the DMP’s review. 
b. Annual change in the number of DMP Preferrals
(By type of investigation: NIS, MP, Unit)
N/A
c. Ratio of DMP Preferrals compared to total number of files referred to DMP
(By type of investigation: NIS, MP, Unit)
IncreasingFootnote 10
7. Frequency of Courts Martial
(JAIMS)
a. Number of Courts Martial (CM) completed in year
(By type of offences, and by type of CM: General CM; Standing CM)
N/A To put outcomes into context and to be able to relate fluctuations in outcomes with variations in inputs and activities, the PMF includes measures of inputs. The frequency of courts martial is relevant to understand the volume of work moving through the court martial system.
b. Annual change in the number of CM completed in year as compared to previous year, expressed as a percentage
(By type of offences, and by type of CM: General CM; Standing CM)
N/A Recognizing that intangible aspects effect the length of courts martial, monitoring the average length of CMs provides information on the relative complexity of trials and when compared with other indicators provides insight into the efficiency of the court martial process.
c. Average length of CM (in sitting days) N/A
8. Cost of Courts Martial
(Admin DataFootnote 11 and JAIMS)

 

a. Average cost of OCMJ per trial (total annual expenditures / # of CM) Decreasing This indicator examines the efficiency of the court martial process by measuring the productivity of its three constituent components in relation to their expense. The costs of courts martial depend on many factors. This indicator strives to provide generalized data relating to the costs of the three principal organizations responsible for delivering a courts martial system for the CAF.
b. Average cost of Prosecution per trial (total annual expenditures / # of CM) Decreasing
c. Average cost of Defence Counsel Services per trial (total annual expenditures / # of CM) Decreasing
DIMENSION (Source) INDICATORS DESIRED TREND DIRECTION RATIONALE
MJS Outputs
9. Timeliness of process
(JAIMS)
a. Average amount of time (in days) elapsed between date of first reporting of alleged offence and date of charges being laid in the MJS
(Excludes cases greater than 1 year, which will be individually reported)
Decreasing Timeliness of process, including the time between the reporting of an alleged offence and the date charges are laid, is important to an effective, efficient, and legitimate military justice system. This indicator will monitor the timeliness of the MJS during the pre-charge period. Starting the clock from the date of first reporting, rather than the date of the offence will avoid skewing the indicator with historical reporting which can be decades after the date of the alleged offence.
b. Average amount of time (in days) elapsed between charges being laid and case final resolution
(By type of offences and type of service tribunal, for all service tribunals completed during the reporting year)
Decreasing

Timeliness of process is important to an effective, efficient, and legitimate military justice system. Accused have a constitutional right to be tried in a reasonable amount of time.

ST must commence within 1 year of date of alleged offence (unless limitation period waiver by accused).

CMs have been found to be subject to the SCC decision in Jordan and are subject to a presumptive ceiling of 18 months from the date of charge to the actual/anticipated end of the trial.

This indicator examines the efficiency, effectiveness, and legitimacy of the MJS by measuring the timeliness with which cases are processed through the system.

c. Percent change in average amount of time (in days) between charges being laid and case final resolution compared to previous year
(By type of offences and type of service tribunal)
N/A
d. Average age of all pending cases within the MJS on 31 March each year
(in days, from date of charge)
Decreasing
e. Percent change in the average age of all pending cases compared to previous year N/A
f. Percent of cases in which one or more time standards is not met Decreasing
10. Pre-trial detention
(JAIMS)
a. Number of pre-trial detention periods exceeding 5 daysFootnote 12 N/A This indicator points to legitimacy (and fairness) issues, particularly if one relates this indicator to the one on delays (age of case).
11. Early resolution of cases
(JAIMS)
a. Percentage of cases in which a guilty plea to all charges is entered before a Court Martial N/A This indicator examines the system’s ability to identify the people responsible for service offences, gather sufficient evidence to support a reasonable prospect of conviction at CM, and process the case for adjudication in a timely manner. This indicator also provides information on the “quality of the decisions” to prosecute (efficiency), as well as whether the decisions are legitimate.
b. Percentage of cases in which prosecutor decides not to proceed (withdraw) with all charges(s) preferred by DMP N/A This indicator helps understand the appropriateness of charging decisions and provides insight into potential over-charging. It also provides a measure of the practice of plea bargaining in the MJS and provides insight into the effectiveness of the early resolution process.

12. Outcomes of proceedings
(JAIMS)

a. Percentage of convictions at ST
(finding of guilt on one or more charge(s), or acquittal on all charges)
N/A This indicator provides information on the “quality of the decisions” to charge and prosecute (efficiency), as well as whether the decisions are legitimate.
b. Percentage of convictions at CM
(finding of guilt on one or more charge(s), or acquittal on all charges)
N/A

13. Legal assistance provided to accused
(JAIMS)

a. Percentage of accused charged who are represented or receiving other forms of legal assistance from DDCS lawyer
(By gender, type of offence, type of assistance, and type of proceedings)

Increasing This indicator provides information on the fairness of the system to accused members, their rates of access to justice, and accused members perception of effectiveness of representation provided by DDCS.

14. Summary Trial Reviews
(JAIMS)

a. Percentage of STs reviewed by Review Authorities Between 3% and 6%Footnote 13 The indicator provides information on the ST system and convicted members’ perception of fairness. It also relates to the quality of decisions made by presiding officers, which are directly related to effectiveness, efficiency, and legitimacy.
15. Number of judicial reviews
(Federal Court) (Admin data)
a. Annual change in the number of cases reviewed by the Federal Court Decreasing This indicator provides information on the ST system and its review process which is directly related to its effectiveness, efficiency and legitimacy.
16. Number of appeals to CM Appeal Court of Canada
(JAIMS)
a. Annual change in the number of cases appealed to the CMAC Decreasing This indicator provides information on the CM system and is directly related to its effectiveness, efficiency and legitimacy.
DIMENSION (Source) INDICATORS DESIRED TREND DIRECTION RATIONALE
MJS Outcomes
17. Outcomes of ST Reviews
(JAIMS)
a. Percentage of ST Reviews in which the Review Authority upholds the Presiding Officer’s decision Increasing This indicator provides information related to the quality of decisions made by presiding officers which is directly related to the effectiveness, efficiency, and legitimacy the ST system.
18. Outcomes of judicial reviews
(Admin Data)
a. Percentage of Judicial Reviews in which Federal Court denies request (Applicant’s request entirely denied versus allowed in whole/part or) by type of review (ST, Review, Other) Increasing This indicator provides independent and impartial information on the effectiveness and legitimacy the ST system and its review process.
19. Outcomes of appeals to Court Martial Appeal Court
(JAIMS)
a. Percentage of appeals to CMAC in which the appeal is entirely dismissed
(Appellant’s appeal entirely dismissed versus allowed in whole/part) 
Increasing This indicator provides independent and impartial information on the effectiveness, efficiency, and legitimacy the CM system.
20. Access to justice for accused
(Survey data)
a. Percentage of accused/convicted members reporting satisfaction with their level of access to legal assistance
(By gender, rank, type of offence, and component)
Increasing This indicator provides information on the fairness of the system to accused members, their rates of access to justice, and accused members perception of effectiveness of representation provided by DDCS.
b. Percentage of accused/convicted members reporting satisfaction with the CAF’s procedural/administrative handling of their case
(By gender, rank, type of offence, and component)
Increasing This indicator provides information on members’ perception of procedural fairness and relates to morale and discipline of the CAF.
21. Access to justice for victims of service offences (Survey Data) a. Percentage of victims of services offences reporting satisfaction with their level of access to legal assistance/information (By gender, rank, type of offence, and component) Increasing This indicator provides information on the effectiveness of the system as it relates to victims by assessing their rates of access to justice, through access to legal assistance and information.
b. Percentage of victims reporting satisfaction with the CAF’s procedural /administrative handling of their case
(By gender, rank, type of offence, and component)
Increasing This indicator provides information on victims’ perception on the CAF’s ability to effectively and efficiently administer the MJS and relates to morale and efficiency of the CAF.
22. Confidence in the MJS among Victims (JAIMS + Survey data) a. Percent annual change in the number of alleged offences reported by victimsFootnote 14   in year
(By gender, rank, type of offence, and component)
N/A This indicator provides information on victims’ confidence in the MJS, their reporting behaviour, and is a measure of the credibility of the system and possibly a proxy measure for the “legitimacy” of the system.
b. Percent annual change in the number of alleged offences reported (other than by victims)Footnote 15   in year
(By gender, rank, type of offence, and component)
N/A This indicator provides information on the confidence in the MJS, members’ reporting behaviour, and is a measure of the credibility of the system and possibly a proxy measure for the “legitimacy” of the system.
c. Percent of victims expressing confidence in the MJS and change over previous year. Measured using a five-point scale. (By gender, rank, type of offence, and component). Increasing levels of confidence
23. Confidence in the MJS among Commanding Officers (Survey data) a. Percentage of commanding officers expressing confidence in the MJS (to promote discipline, efficiency, and morale) and change over previous year. Measured using a five-point scale.
(By gender, rank, environment, and component).
Increasing levels of confidence This indicator provides information on the reported confidence in the MJS by the leadership of the CAF and is a measure of the credibility of the system and possibly a proxy measure for the “legitimacy” of the system.
24. Confidence in the MJS expressed by CAF members (Survey data) a. Percentage of CAF members expressing confidence in the MJS and change over previous year. Measured using a five-point scale.
(By gender, rank, environment, and component).
Increasing levels of confidence This indicator provides information on the reported confidence in the MJS by the members of the CAF and is a measure of the credibility of the system and possibly a proxy measure for the “legitimacy” of the system.
25. Perceived fairness of the MJS (Survey data) a. Changes in the perceived fairness of the MJS among victims. Using a five-point scale.
(By gender, rank, type of offences, and component).
Increasing levels of fairness This indicator provides information on the perceived fairness of MJS by accused/convicted members, victims of service offences, and other MJS participants, and is a measure of the effectiveness of the system and possibly a proxy measure for the “legitimacy” of the system.
b. Changes in the perceived fairness of the MJS among accused/convicted members. Using a five-point scale.
(By gender, rank, type of offences, and component).
Increasing levels of fairness
c. Changes in the perceived fairness of the MJS among MJS participants (Commanding Officers, Presiding Officer, Assisting Officers, and Witnesses). Using a five-point scale.
(By gender, rank, environment, and component).
Increasing levels of fairness
26. MJS contribution to Efficiency in Canadian Armed Forces

To be developed.
(This is a purpose of the MJS). 

  Further work, and perhaps experimentation, will be required to determine how to measure the contribution the MJS to the efficiency of the Canadian Armed Forces. One or more valid and reliable proxy indicators should be identified. 
27. MJS contribution to Discipline in the Canadian Armed Forces 

To be developed.
(This is a purpose of the MJS). 

  Further work, and perhaps experimentation, will be required to determine how to measure the contribution the MJS to the discipline of the Canadian Armed Forces. One or more valid and reliable proxy indicators should be identified.
28. MJS contribution to Morale in the Canadian Armed Forces To be developed.
(This is a purpose of the MJS). 
  Further work, and perhaps experimentation, will be required to determine how to measure the contribution the MJS to morale within CAF. One or more valid and reliable proxy indicators should be identified.

 

Throughout the lifecycle of the PMF, it should be periodically confirmed that the most valid and reliable indicators have been identified, and that any additions, deletions, or replacements be considered where necessary.

 

.-.-.-.-.

Back to Top

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: