Letter to the Editor - HPCDP: Volume 36-5, May 2016
Volume 36 · Number 4 · April 2016
Letter to the Editor
December 9, 2015
To the editor:
I call your attention and that of your readers to several serious concerns I have about much of the content in the journal's last special issue, the focus of which was child maltreatment. Most of the papers drew heavily on data in The Canadian Incidence Study of Reported Child Abuse and Neglect (CAN) - CIS. Unfortunately, CIS is based entirely on reports of suspected abuse sent to child welfare agencies. None of the papers, or the editorials, fully disclose or sufficiently emphasize the important shortcomings of this database. A report in it is often treated as equivalent to a proven case. There is no provision for updating or modifying the information if, after 6 weeks a report is dismissed following further enquiry. Sexual abuse is a criminal offense and thus rests on a legal decision that may take years. Even "caseness" requires confirmation after further investigation. Reporting as a proxy for actual cases is unlikely to be valid and either overestimates or underestimates the number of true cases. Admittedly, most limitations of concern relate to the database, not the analyses. Although CIS is not a population-based health study but rather a study of the social service response to possible child maltreatment, all such limitations greatly matter if these data are used as a basis for making preventive policy. The time is long overdue for those involved in this complex issue to agree on a sensible criterion that lies somewhere along the continuum from simple reporting to criminal conviction.
The CIS website reveals that only 27% of investigated cases remained open after initial investigation and this proportion actually decreased over time.Letter reference 1 Applications to child welfare court only involved 5% of 223 063 cases in 2008. In that year, 36% of investigations were substantiated, 30% were unfounded and 8% had insufficient evidence. In the section on the website where data collection and verification procedures are described, what is reported is troubling:
"Data collected in 2003 … was compared with information in the case files from one of the larger CIS-2003 sites. While there was good correspondence on many items, it became apparent that despite specific instruction … to include only investigations of child maltreatment, a number of cases that appeared to involve only concerns about future risk had been coded as maltreatment investigations."Letter reference 2 The only other reference to validation describes the use of focus groups for feedback on the data collection tool. Reliability testing was, apparently, equally problematic. Accordingly, what is in the database is a mixture of suspected cases—including many that will not be investigated further—some that will be handed over to the police and some that ultimately may be confirmed. Treating all of these as equivalent is hardly a sensible basis for making policy.
I also assume reports are far more likely to be sent regarding low-income families than those in the upper-income brackets—compounding these serious problems. Moreover, even if a report results in a charge regarding the behaviour of middle class parents or caretakers, the likelihood of a guilty verdict is probably much less than it would be for families with fewer resources. What this means for scientists—and I hope for policy makers as well—is that the CIS database is biased against the poor. It also may over-represent supposed cases arising in marital disputes, and it certainly over-represents Aboriginal families.Letter reference 3 Fortunately, this study acknowledges the bias and offers much-needed cautionary notes.
I was pleased to see the balanced commentary by American colleagues, Leeb and Fluke.Letter reference 4 It did not, however, include an appraisal of the strengths and weaknesses of CIS. The closest they come is when they write,
"CAN cases may be reported to a wide variety of sentinels (e.g. educators, clergy, physicians, law enforcement, child welfare), or may not be reported to any official source at all. Social stigma and unintended consequences of reporting, as well as cultural and political barriers, also impact reporting both within communities and globally."Letter reference 4,p138 They add,
"Efforts to develop more uniform definitional approaches for administrative sources have been made, but despite these attempts, it remains challenging to proffer accepted definitions in complex policy environments with multiple inter-sectorial stakeholders."Letter reference 4,p139 The commentary also asserts,
"… we and others (e.g. Thacker) suggest that surveillance systems in CAN are incomplete if they do not strive to achieve some degree of insight about the data collection methods, the process of surveillance, or the underlying risk conditions."Letter reference 4,p139 Clearly, the key message here is that readers fully understand how the data are collected and analyzed.
Accordingly, I was much less pleased to read the editorials by Tonmyr.Letter reference 5,Letter reference 6 She is the senior author of three papers she cites as supporting CIS's integrity and validity. Tonmyr neglects to remind readers of the point made by the commentators Leeb and Fluke:
"How you count matters. Relying on a single source of information such as cases reported to child welfare agencies … provides only one view of the problem."Letter reference 4,p139 She also neglects to cite a reference that provides key details about CIS methods.Letter reference 7
Potter et al.Letter reference 8 mention some of the problems with CIS. They state,
"CIS data only include children … reported to and investigated by child welfare agencies. Thus, selection bias may impact the population of children identified in the CIS as some children who experience abuse or neglect (for example, children from low-income families) may be more likely to be the subject of child welfare reports than others."Letter reference 8,p127 These important reminders should appear in each of the papers in this issue, not only in this one. As well, someone should have emphasized other limitations of CIS, notably that it only tracks information during the first 6 weeks of case activity. Furthermore, the CIS
"…does not include […] cases that were investigated only by the police and cases that were never reported [to either the police or child welfare authorities]."Letter reference 2. Finally, the assessments provided by the child welfare workers could not be independently verified.
Leeb and Fluke conclude,
"CAN surveillance remains a challenge for epidemiologists worldwide."Letter reference 4 Unfortunately, this understatement is compounded by what follows:
"But insights derived from our collective efforts, including the CIS, will help target our prevention efforts in the face of this global public health problem."
I am not persuaded this has been established. Hence, until the flaws in CIS are fully disclosed and widely appreciated, the material reported in this issue must be taken with many grains of salt.
Barry Pless, CM, MD, BA, FRCPC, FCAHS, DSc (Hon), FRCPCH (Hon)
Professor Emeritus, Pediatrics and Epidemiology, McGill University
Editor Emeritus, Injury Prevention
Associate Scientific Editor, Health Promotion and Chronic Disease Prevention in Canada
Addendum: Publication of the Public Legal Education and Information Service of New BrunswickLetter reference 9
Department of Social Development assesses all reports of abuse and neglect. If social workers suspect that the security and development of the child could be in danger, they will investigate. Where the matter involves criminal offences, the police will join in the investigation. Criminal offences include sexual abuse, serious physical abuse or physical neglect. The investigation could involve separate interviews with the person who reported the abuse, the child, the suspected abuser and other family members if necessary. If the parents prevent the investigation, the Department can apply to Family Court for an order to conduct the investigation or take the child into custody.
The police will consult with the Crown Prosecutor to decide if there is enough evidence to lay criminal charges against the accused. (If the case comes to court this is what happens) -
At a hearing for a child protection order or a criminal trial, each side presents evidence to the judge. In criminal proceedings, the judge will listen to the evidence and then rule on the guilt of the accused. In a hearing for a child protection order, the judge will decide what is in the "best interests" of the child and make the proper order. This order could be taking the child from parents' custody, observing the family for a time, preventing the abuser from contacting the child, or protecting the child from himself.
Guest Editor's response
I thank Dr. Pless for his letter regarding the special double-feature issue focussing on child maltreatment surveillance and prevention. I welcome the opportunity for discussion and closer scrutiny of the points he raises.
I regret that Dr. Pless feels that the issue lacks sufficient documentation about the limitations of the Canadian Incidence Study of Reported Child Abuse and Neglect (CIS). The data pertain to investigations conducted after reports are received by child welfare agencies, a fact that is repeatedly stated in all of the publication's papers relating to the CIS. The process of capturing information four to eight weeks following the receipt of the report to child welfare is also clearly described, as are other limitations.Reference 1,Reference 2,Reference 3 Contrary to Dr. Pless's statement, all of the papers cite the CIS website as an additional source of methodological information, including specific shortcomings of the data.Reference 4
As is well known by child maltreatment epidemiologists, and alluded to by Leeb and Fluke,Reference 5 particular challenges impede the accurate and complete measurement of child maltreatment. In Canada, as elsewhere, issues including inconsistencies in legislation and definitions, the myriad of sectors involved in case reporting and social and cultural impediments to disclosure, among others, have been the subject of vigorous discussion since the establishment of the CIS. Through the efforts of the CIS, substantial progress in overcoming these obstacles has been made. For example, consistent definitions governing CIS data collection have been implemented across agencies, despite the variety of administrative approaches to case ascertainment. As Dr. Pless correctly states, the coding of risk investigation was problematic in CIS-2003, but improvements to data capture were introduced in 2008 so that risk is now categorized separately.Reference 4
Regarding the validity of CIS data, several issues—including stigma, cultural practices and fear of disclosure—impede rigorous assessment of the occurrence of child maltreatment. Canadian research shows that estimates of child maltreatment prevalence are lower when based on reports from child protection workers than self-reports. With respect to reliability, inter-rater reliability testing is complicated by the fact that any two child protection workers rarely have access to the same information about an allegation of maltreatment, so comparisons of their responses are meaningless. However, a test–retest of reliability (3.0–4.5 weeks) of the CIS coding found substantial to excellent results on a majority of variables.Reference 6
As acknowledged by Potter et al.,Reference 1 and as Dr. Pless correctly reiterates, selection bias probably results in over-representation in the CIS of children from certain subpopulations, including those of low socioeconomic status (SES), who are more likely to come to the attention of child welfare officials. This is not to diminish the importance of poverty in the study of child maltreatment. Indeed, the link between low SES and a variety of negative health and well-being outcomes is well recognized.
I take issue with Dr. Pless's implication that maltreatment surveillance should be restricted to cases that have been "proven" in court. He contends many groundless reports are filed in the context of marital disputes. However, it is widely acknowledged that cases of child maltreatment in the CIS represent "the tip of the iceberg." Traditionally, surveillance methodologies including sentinel and self-report studies do not include a consideration of legal status. Furthermore, cases of maltreatment or suspected maltreatment that are not brought to the attention of the courts may involve families who would benefit from the available services.
Dr. Pless expresses reservations related to the usefulness of CIS data to policy- and decision-makers. As mentioned in my editorial, many professionals are involved in protecting children from harm, but provincial and territorial child welfare legislation authorizes child welfare organizations to investigate allegations of child maltreatment. In addition, the legislation stipulates the conditions under which children may be removed from parental care. Thus, reports to child welfare provide a valuable source on which to base policy and programming decisions, especially regarding child welfare. Research has also demonstrated the utility of the reports to a wide variety of other sectors.Reference 7
Despite their limitations, CIS data are of value in informing policy and practice. For instance, risk investigations identify children at risk of maltreatment and thereby provide the basis for preventive intervention. Previous research indicates that many children are reported to child welfare agencies repeatedly. In Canada, the risk of recurrent maltreatment could be mitigated by improving family and contextual circumstances; data from the CIS provide a unique source of information on the conditions most strongly associated with maltreatment and repeated referrals.
Self-reported survey data, though not without their own limitations, would provide another important source of information. Accordingly, the Public Health Agency of Canada (PHAC) continues to partner with Statistics Canada to explore ways to include child maltreatment questions in population-based health surveys. Linked administrative databases offer yet another potential source of data on child maltreatment and is one of several options for future child maltreatment surveillance within PHAC.Reference 8,Reference 9
Again, I appreciate the opportunity to respond to this letter, which touches on some of the challenges to the study of child maltreatment and is particularly timely as PHAC moves toward a new cycle of CIS data collection in 2018. Accurate surveillance of the health and well-being of Canadians is a key objective of PHAC, and we value the perspective of those who share this vision.
Lil Tonmyr, PhD
Surveillance and Epidemiology Division, Public Health Agency of Canada
Report a problem or mistake on this page
- Date modified: