Background
A common method of measuring the predictive performance of models was described by Sheiner and Beal in 1981. They suggested measuring predictive performance by expressing bias and imprecision of the model by calculating the mean prediction error (MPE) and the root mean squared error (RMSE). There are occasions where the 95% confidence interval of MPE includes zero (which indicates unbiased prediction) but the model incorrectly predicts at certain regions. In this case, the MPE approach is insensitive to detect systematic bias. A loess line can be fitted on the observed vs predicted plots, however this only gives a qualitative measure of systematic bias.
The objective of this study is to expand the MPE method to detect systematic bias. A motivating example used in this analysis is based on our recent study on the ability of a dosing tool for warfarin to predict maintenance dose.
Methods
The method of Sheiner and Beal is a single bin approach where all observations are combined for computation of MPE. We propose a general case where we extend their method to an asymptotic infinite number of bins. In this approach the second derivative of the MPE over the bins provides an estimate of the rate of systematic bias and the asymptotic properties provides an estimate of the absolute bias. This method is illustrated for a warfarin INR vs dose example. The observed vs predicted maintenance doses for 46 patients were plotted. The MPE for the overall predictions were calculated (1 bin approach). We then binned at equidistant locations on the x-axis and the slope of the MPE vs bins were computed for each subsequent set of bins (from 2 bins up to 12 bins). We then plotted the slopes vs number of bins and fitted with a line of best fit.
Results
The single bin approach produced unbiased dose predictions 0.36 mg/day (95% CI -0.15, 0.89). Visualisation from the loess line of the observed vs predicted dose plots showed that the positive bias occurs mainly at higher dose ranges. The slope of MPE vs bins declined exponentially from 2 to 12 bins. The coefficient, θ, of the second derivative ∝exp(θ.b) of the MPE over the bins was greater than zero indicating that a systematic bias was present.
Conclusion
This method can be easily automated and provide a statistic for determining the presence of systematic bias. This was illustrated in our motivating example with warfarin dose predictions.
References:
1. Sheiner LB and Beal SL Some suggestions for measuring predictive performance Journal of Pharmacokinetics and Biopharmaceutics. 1981;9:503-512.
2. Saffian SM, Wright DFB, Roberts RL, Duffull SB, Methods for predicting warfarin dose requirements, Therapeutic Drug Monitoring. 2014 – article accepted.