Original quantitative research – Validation of Canproj for projecting Canadian cancer incidence data

Correspondence: Alain Demers, Public Health Agency of Canada, 785 Carling Avenue (603A1), Ottawa, ON  K1A 0K9; Tel: 613-797-2129; Email: alain.demers@canada.ca

Abstract

Introduction: Cancer projections can provide key information to help prioritize cancer control strategies, allocate resources and evaluate current treatments and interventions. Canproj is a cancer-projection tool that builds on the Nordpred R-package by adding a selection of projection models. The objective of this project was to validate the Canproj R-package for the short-term projection of cancer rates.

Methods: We used national cancer incidence data from 1986 to 2014 from the National Cancer Incidence Reporting System and Canadian Cancer Registry. Cross-validation was used to estimate the accuracy of the projections generated by Canproj and relative bias (RB) was used as validation measure. The Canproj automatic model selection decision tree was also assessed.

Results: Five of the six models had mean RB between 5% and 10% and median RB around 5%. For some of the cancer sites that were more difficult to project, a shorter time period improved reliability. The Nordpred model was selected 79% of the time by Canproj automatic model selection although it had the smallest RB only 24% of the time.

Conclusions: The Canproj package was able to provide projections that closely matched the real data for most cancer sites.

Keywords: neoplasms, forecasting, validation studies

Introduction

For the past 30 years, the Canadian Cancer Society and the Government of Canada (Public Health Agency of Canada and Statistics Canada) have published an annual comprehensive report, Canadian Cancer Statistics (CCS). The report includes a series of population cancer incidence and mortality counts and rate projections that fill the gap between the latest available year of data and the year the report is released. These projections are a planning and prioritizing resource for stakeholders; they also keep the Canadian population informed on the considerable burden of cancer.

A few projection models have been used over the years to produce the CCS. The Poisson regression^{Footnote 1} used from 2003 to 2012 changed to Nordpred in 2011/2012. Nordpred, an R-package that was developed in Norway, makes available one single projection model, the age–period–cohort (APC) model with a drift component.^{Footnote 2} Nordpred is a well-studied package that has been shown to improve the reliability of cancer projections.^{Footnote 3Footnote 4Footnote 5Footnote 6Footnote 7}

In an effort to further cancer projections, Qiu et al. developed Canproj, which is also an R-package.^{Footnote 8} Canproj has three key advantages over Nordpred: 1) replacement of the Poisson distribution by the negative-binomial distribution when over-dispersion is present; 2) inclusion of an age–cohort model; and 3) a set of hybrid models that combine the strengths of Poisson or negative-binomial regression, the segmented regression method,^{Footnote 9} and an average method for projections based on age-specific counts. Some of the features of Canproj were used for the 2017 CCS^{Footnote 10} while the full package was utilized for the 2019 CCS.

Canproj is a relatively new cancer-projection tool that has neither been extensively used nor validated.^{Footnote 11Footnote 12} The objective of our project was to validate the national short-term (up to 5 years) cancer incidence projections generated by the Canproj package using Canadian data. Specifically, we compared the outputs of the Canproj projection models to actual data using the holdout cross-validation method^{Footnote 13} and graphical representation. We also evaluated the automatic model selection features of Canproj (decision trees) to assess the capacity of these functions to select the best model.

Methods

Data

Cancer incidence data from 1986 to 2014 from the National Cancer Incidence Reporting System (NCIRS) and Canadian Cancer Registry (CCR) were used for the analysis.^{Footnote 14} Data from the province of Quebec were not included since the provincial cancer registry has not submitted new data to the CCR since 2010. The data file used the International Agency for Research on Cancer’s international rules for multiple primary cancers.^{Footnote 15} Results were tabulated for all cancers combined, by cancer site (the same cancer sites as those included in the CCS annual reports) and sex.^{Footnote 10} A dataset was created for each combination of sex (n = 2) and cancer type (19 common to males and females plus five sex-specific types: cervix, ovary, uterus, prostate and testis). Datasets contained information by years from 1986 to 2014 and eighteen 5-year age groups (0–4, 5–9, …, 85+). Annual population estimates by geography, age and sex were provided by Statistics Canada with post-censal population estimates based on the 2016 Canadian census.^{Footnote 16} Inter- and post-censal estimates were adjusted by Statistics Canada for net under-coverage. Rates were age standardized using the direct method and the 2011 Canada population.^{Footnote 17}

Canproj

The Canproj R-package contains several models used to project cancer incidence or mortality data. These include the Nordpred model, which incorporates age, drift, period and cohort effects; the age–cohort model; three hybrid models that incorporate age and potentially period effects (age-specific or all ages); and the 5-year average model (Table 1).^{Footnote 18}

Table 1. Models available in the Canproj R-package and variables included in the models

The Canproj package uses two decision trees to determine which model is the most appropriate based on the significance of the variables. Alternatively, models can be selected individually. At first, Canproj considers four variables, namely age, period, cohort and a drift parameter; this is the most complex model, and these are the variables Nordpred uses. Canproj first determines if the cohort variable is significant. If it is significant, Canproj determines if the drift parameter is significant. If the cohort variable and the drift parameter are both significant, Canproj selects the Nordpred model to make the projections. If the cohort variable is significant but the drift parameter is not, Canproj selects the age–cohort model.

If the cohort effect is not significant, Canproj selects one of the hybrid models. If the number of cases is too small to run a regression model, a 5-year average is calculated. If the number of cases is big enough, Canproj will fit two models: an “age-common trend” model and an “age-specific trend” model. If the age-specific trend model has a better fit, then this model is selected. If not, the age-common trend model is selected. The slope of the common trend variable is then tested to determine if it differs from zero. If it is not different, then only the age variable is used in the model; if it is, the age + common trend model is used.

Validation

Cross-validation was used to estimate the accuracy of the Canproj-generated projections by using a subset of the data (the training data) and validating the results on the other subset (the independent testing data). This study used the holdout method^{Footnote 13} to create the training and the independent testing datasets. Data from 1986 to 2010 (five 5-year periods) were used as the training data, and data from 2011 to 2014 (the last 4 years of data) were used as the independent testing data. The predictions from the training model and the actual data from the last 4 years were compared to evaluate the accuracy of the projection models.

The validation measure we used, the relative bias (RB), compares the expected value generated by the projection models to the observed values in the testing dataset for diagnosis years 2011 to 2014. The RB measures the relative difference in percentage between the expected (or projected) value (E) and the observed value (O).

$R{B}_t=\frac{\mid E_t-Q_t\mid }{O_t}\times 100$ , where t = 2011 to 2014

In our case, the “value” investigated is the age-standardized rate.

The RBs were summarized by projection model, cancer type and sex.

We compared the mean and median RBs by model, cancer type and sex over the 4-year projected (testing) period. Median RB indicates the typical performance of a model, whereas mean RB (due to its sensitivity to extreme values) helps reveal models that are typically accurate but occasionally very inaccurate.

Joinpoint analyses

We used Joinpoint Trend Analysis Software version 4.5.0.1 (National Cancer Institute, Bethesda, MD, USA)^{Footnote 19} to calculate trends in Canadian cancer incidence by type and sex between 1986 and 2010. Joinpoint model estimates were used to calculate the 1986 to 2010 RBs. This measure gives an estimate of the variability of the training data, which we compared to the RB measured on the projected data. The maximum number of joinpoints was set to 4; the minimum number of observations from a joinpoint to either end of the data was set to 3; and the minimum number of observations between two joinpoints was set to 4. Otherwise, the default joinpoint parameters were used. The log-transformed age-standardized rates and associated standard errors input into joinpoint were calculated in statistical package SAS version 9.3 (SAS Institute Inc., Cary, NC, USA). Canproj was run using R version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria) and RStudio version 1.1.453 (RStudio Inc., Boston, MA, USA).

Performance indicators

Two indicators were used to highlight which models would likely project rates less reliably. The first was the identification of a joinpoint over the most recent 10-year period in the data used to train the projection models (2001–2010). Recent changes in the trend could indicate that the models will have more difficulty performing reliable projections. We divided the joinpoints between those that happened between 10 to 6 years before the last year of training data available and those that happened 5 to 3 years before the last year of data. Joinpoints were not allowed to occur between 0 and 3 years. In Table 2, yellow cells indicate joinpoints that happened between 2001 and 2005 and orange cells indicate those that happened between 2006 and 2008.

Table 2. 2011–2014 median relative bias (%) by model and cancer type

For the second indicator, we used the RB ratio, which is the ratio of RB from 1986 to 2010 to the RB for the 2011–2014 period. We considered that the bias from the projected rates should be at least equal to or greater than the bias in the rates that were used to build the projection models. To obtain the 1986 to 2010 RB, we used the output of the joinpoint analysis. In Table 2, if the 2011–2014 RB was 2 to 5 times higher than the 1986–2010 RB, table cells are in yellow; if the 2011–2014 RB was more than 5 times higher than the 1986–2010 RB, the cells are in orange. These cutoffs were arbitrarily determined after looking at the distribution of the results.

Results

Canproj models

Five of the six models (Nordpred, the age–cohort model, the hybrid common trend model, the hybrid age-specific trend model and the 5-year average model) had mean RB between 5% and 10% and a median RB around 5% (Figure 1). Greater variation was observed in the mean and median RB when the accuracy of the projection models was compared by cancer site (Figure 2). None of the models were good at predicting prostate cancer and a greater predictive variability was apparent for cancers of the thyroid, larynx, bladder, liver and brain/central nervous system (CNS).

A more detailed and slightly different picture emerges when models are graphically compared by type of cancer and sex (Figure 3, Table 2). The performance of all projection models was poor for male all cancer sites combined, male and female colorectal, prostate, male bladder, male and female brain/CNS, female Hodgkin lymphoma and male myeloma. The greater variation observed in Figure 2 for cancers of the liver, larynx and thyroid seems to be due to the inability of a few models to predict rates.

Cancer sites that showed recent change in trend for which projections could potentially be improved by changing the length of the data included male all cancers, male and female colorectal cancer and prostate cancer. We ran separate hybrid models on these cancers sites using the last 7 years of data only. It was possible to increase the fit of the projections substantially for all four cancer sites. For colorectal cancer, it was possible to bring the RB ratio from 16.0 to 2.6 for males and from 5.5 to 1.9 for females. We were also able to bring the RB ratio from 5.7 to 5.4 for male all cancers combined and from 11.2 to 7.7 for prostate cancer.

Canproj model decision trees

As shown in Table 3, the cohort effect and the drift parameter were significant 79% of the time (34 out of 43 models), which makes Nordpred the model most often selected by Canproj. However, Nordpred was the model with the smallest RB only 24% of the time. Nevertheless, the mean RB was between 0 and 5% for at least one of the six models 76% of the time and between 6% and 10% for at least one model 20% of the time.

Table 3. Canproj decision tree: average relative bias by model, sex and cancer site

Discussion

Our aim was to validate short-term projections generated by Canproj using Canadian cancer incidence data. The results show that the range of models Canproj offers supports making reliable projections for most of the cancer sites investigated. When variations in rates were identified within the last 10 years of training data, it was possible to use the recent, shorter time period as the projection base for the hybrid models to improve the accuracy of the projected rates.

The large jump in bladder cancer rates in 2013/2014 is due to changes in reporting rules in Ontario;^{Footnote 20} starting in 2013, Ontario added in situ bladder to malignant bladder cancer in their registry.

Brain/CNS, colorectal cancer, female Hodgkin lymphoma, prostate and male all cancer combined rates are declining faster than the models predicted, while male myeloma is increasing faster than the models predict. The poor performance at predicting these cancer rates is related to the recent and rapid changes in their rates that were not part of the training dataset or happened in the last few years of the training dataset.

We evaluated the automatic model selection feature of Canproj (decision trees) to assess the capacity of these functions to select the best model. For the national dataset, Nordpred was the model most often selected by Canproj decision tree although it was the one with the smallest RB only 24% of the time. Other models can outperform Nordpred when analyzing data from smaller populations.^{Footnote 21} Personal and others’ experiences with Canproj suggest that the decision tree selection should be used in combination with individual outputs of each model and expert advice to select the best projection model.^{Footnote 22}

The results of this project build on prior Canadian studies that examined different cancer projection methods. Lee et al. (2011) compared the accuracy of 16 models and model variations for projecting short-term cancer mortality rates.^{Footnote 23} They found that no single method was able to consistently provide accurate forecasts for a wide range of cancer sites and that a choice of models is preferable. Qiu et al. (2010) compared the Nordpred model, the generalized additive model and the Bayesian model.^{Footnote 8} They concluded that when the age, drift and cohort effects are present, the Nordpred method is the preferred approach; when the age and cohort effects are present, an age–cohort model is the best approach; and when the cohort effect is not present, a hybrid method should be used. They also found that for small cancer sites, data aggregation is required to apply the hybrid method. In 2010, the Canadian Cancer Projections Network (C-Proj) released a report in which they evaluated Nordpred, hybrid, age–cohort and Bayesian models using Markov chain Monte Carlo cancer incidence projection methods with data from the Nova Scotia Cancer Registry.^{Footnote 21} They suggested that the age–cohort method should be used for cancer projections for provinces with small and stable populations.

Although cancer incidence projections are routinely performed, only a few studies describe the evaluation of alternative methods; the recommendations depend on the population included and projection time frame. Stock et al. (2018) used a Bayesian approach to project cancer incidence rates to 2030 using data from the German cancer registry.^{Footnote 24} They found that this method offered advantages in terms of flexibility, interpretability, transparency and level of detail, but they did not recommend using it for short-term data. Pesola et al. (2017) compared a number of models (null, age–drift, age–period, age–cohort and APC) to predict pediatric and adolescent cancer incidence in England to 2030.^{Footnote 25} The model fit results showed that the age–drift model offered as good a fit to the data as more complex models for all cancers in children. An APC model with natural cubic splines was evaluated when predicting cancer incidence and mortality in the United Kingdom until 2035.^{Footnote 26} The basis of the APC model is that past trends will continue into the future. If vaccines or new treatments that change cancer incidence and mortality are developed, the model will not anticipate these changes, reinforcing the importance of using recent data and completing projections at regular intervals.^{Footnote 27} Katanoda et al. (2014) examined three projection models’ ability to project short-term cancer incidence in Japan: generalized linear model with age and period as independent variables (A+P linear); generalized linear model with age, period and their interactions (A*P linear); and generalized additive model with age, period and their interactions smoothed by spline (A*P spline).^{Footnote 28} They used Nordpred in their preliminary analysis and it failed to predict the peak in liver cancer in the mid-1990s.

Strengths and limitations

This project has several limitations. In all the models Canproj uses, the variables age, period and cohort encompass all the changes and improvements in risk factors, demography and ethnic profile of the population, prevention, early detection and treatment. More details on these cancer rate determinants would improve the capacity for making more reliable projections. However, the level of information needed may be hard to obtain in some jurisdictions and, for most of the cancer sites investigated in the project, the age, period and cohort information has proven sufficient for making reliable projections.

We did not conduct a detailed Canadian provincial data analysis in the present exercise, but we expect that as provincial populations get smaller, models other than Nordpred would become the most frequently selected through the decision tree.

The data used in the models did not include data from the province of Quebec and consequently does not represent the entire country.

Finally, as with all methods, projections rely on the assumption that past trends will continue into the future, which may not always be the case.

Conclusions

Health care planners and policy makers need to know about the future burden of cancer to help them prioritize cancer control strategies, allocate resources and evaluate treatments and interventions. The Canproj package can provide reliable cancer projections to help them support their task.

Conflicts of interest

The authors have no conflicts of interest to declare.

Authors’ contributions and statement

AD, ZQ and AS were involved in the design and conceptualization of the work.

AD and ZQ were in involved in the analysis of the data.

AD drafted the paper.

All authors provided input for the interpretation of the results and revision of the paper.

The content and views expressed in this article are those of the authors and do not necessarily reflect those of the Government of Canada.