Missing confidence intervals

[KirstenBos]

Dear Professor Jackson,

I am currently conducting a msm analysis to identify predictors of transitions from living at home to hospital readmission, recovery, or death. While doing so, we are encountering limitations related to the complexity of our data when using the msm package.

We deliberately chose the msm framework because the timing of recovery was assessed only at fixed time points, meaning that the exact recovery times are unknown. For the other transitions, however, we do have exact transition times, and these have been specified using the obstype argument, as described in your documentation.

In the attached figure, we show all transitions observed in our data. Our primary interest is in identifying predictors for transitions from state 3 to states 4, 5, and 6.

My first question concerns the feasibility of modeling these transitions: in our data, the number of observed transitions from state 3 to state 6 is only 10. In your opinion, is this number sufficient to obtain reliable estimates, or would you recommend excluding this transition from the model?

In addition, we are struggling with the optimal use of the msm model and the interpretation of its results. Using the script attached, we fit models with several covariates applied to all transitions (all covariates are added seperatly). However, for some covariates we do not obtain confidence intervals for the hazard ratios, which is problematic for reporting our results in a publication.

My second question is therefore what the exact cause of these missing confidence intervals might be, and how this could potentially be resolved. I have already tried changing the optimization method to "BFGS" and "CG" (Conjugate Gradient), but in those cases some covariates fail to return any results at all.

I would very much appreciate your advice on these issues.

Kind regards,
Kirsten Bos

run_msm_single_cov.txt

[chjackson]

If there are no CIs, I would not trust the fitted model at all, even the point estimates, because it hasn't identified the maximum likelihood. Usually this is because the likelihood is flat with respect to these parameters, and you should simplify the model. Sometimes tweaking the optimisation can work, but it's much more common that the model is too complex to be informed by the data.

If there's only 10 deaths from a state, then I would not expect to be able to build a reliable model for predictors of death from that state. So first thing I'd suggest is to remove the covariate effects on 3-6.

[KirstenBos]

Dear Professor Jackson,

Thank you for your response. I also ran the model without the covariate effect on 3–6, but unfortunately I still only got an HR with a corresponding CI for six out of the 18 covariates.

Is there anything I can change in the model to make it less complex? Or tweaking the optimisation?

Currently, we are including the following covariates: age (continuous); sex (binary); clinical frailty scale (continuous); Charlson comorbidity index (continuous); modified early warning score (continuous); number of home medications (continuous); frailty index (continuous); grip strength (continuous); daily functioning pre-admission (continuous); length of stay (continuous); complications (binary); home care (binary); and living alone (binary). CI is calculated for age, gender, clinical frailty scale, Charlson comorbidity index and grip strength alone. Does the likelihood of a CI being calculated depend on the type of covariate? Or does the inability to calculate a CI say something about the effect of the covariate? Does it indicate that the effect is minimal, for example?

[chjackson]

You're trying to estimate the effect of 18 covariates per transition, from not that many more counts per transition. There is an often-quoted rule of thumb for logistic regression models that you need at least 10 data points per covariate effect you want to estimate. This isn't applicable here because it's a different class of models, but it gives a starting point for understanding what you could expect to be able to do with a sample size of this order of magnitude.

Instead of starting with a big model and trimming it down until it works, I'd advise to start with a small model (e.g. no death, no covariates) make sure the results make sense, then build it up as far as the data can handle.

[KirstenBos]

My apologies — we're adding the variables one at a time rather than all 18 at once to avoid overcomplicating things. I started by building a model without mortality or covariates, and that’s currently running. Now, we’re adding the covariates one at a time. However, the model is only working with six of the 18 covariates. That’s where we are now, and I’m wondering how I can best proceed or optimise.

[chjackson]

You can also have different covariates on different states, or constrain that the covariates have the same effect on different transitions.

[chjackson]

Also be guided by exploratory summaries of the covariate values - there may be categories with small counts, that might be merged with others or removed from the model. The lack of CI / non-convergence is nearly always a symptom of not having enough data to inform the parameter.