Background: Assessment of goodness of fit of a model to the data set is essential to ensure the model provides a reasonable description of the events seen. For logistic regression the most common diagnostic used for this purpose is binning the data and comparing the empirical probability of an event in each bin to the model predicted probability for the mean covariate value in the bin. Although intuitively appealing this method, termed simple binning, may not have useful properties for diagnosing model problems when the study is unbalanced.
Aim: To develop graphical diagnostics to assess the fit of logistic regression models.
Study design: Three different types of study designs were considered. Design 1: Studies which were balanced on events (y-axis) and dose (x-axis covariate); Design 2: studies balanced on only events but unbalanced on dose. Design 3: Studies that are unbalanced on both events and dose.
Simulation: Each of the simulated data sets consisted of 500 subjects. The administered dose was the only covariate and could be 0, 1, 5, 10 and 20 units for design 1, and could be any random integer between 0 and 20 for designs 2 and 3. The number of individuals per dose level was equal for design 1. As well the number of events and non-events were also equal for design 1. The data were simulated with the dose being related to the outcome according to and Emax model on the logit scale.
Estimation: All the data sets were estimated using the Emax model (correct model) and a linear model (wrong model) with dose as the only covariate and using a logistic transformation to the probability domain.
Diagnostics: We propose 2 new diagnostics. In the first diagnostic, random binning, the simulated data are randomly binned based on dose or number of individuals per bin. These random bins are used to estimate the empirical probabilities in each bin. The estimated empirical probabilities and model predictions are plotted to visually inspect the model fit. In the second diagnostic, simplified Bayesian marginal model plots , a linear spline is fitted to the data with up to a maximum of two knots. This was presumed the best empirical description of the data. The posterior distribution of the fits of the Emax (correct model) and linear models (wrong model) were then compared to the spline and the level of agreement between them assessed. The above diagnostics are compared with simple binning.
Results: For all designs the proposed diagnostics performed at least as well or better than simple binning. In case of design 1 random binning and simple binning are similar. In case of design 2 and 3 random binning and simplified marginal model plots were superior in assessing the model fit when compared to conventional binning.
Conclusion: Simple binning fails to show model deficiencies in case of designs 2 and 3. In such cases the proposed diagnostics may be used.
1. Pardoe, I. A Bayesin sampling approach to regression model checking. JSTOR; 10, 617-627 (2001).