Access to estimate and confidence intervals for roc.test

[benneely]

Is your feature request related to a problem? Please describe.
Would like access to the estimate and confidence intervals when running:

roc.test(roc1, roc2, method="delong")

Currently, the estimate returned is a repeat of roc1 and roc2 and no confidence interval is provided.

Describe the solution you'd like
It looks like the proper way to accomplish this might be to add a function like pROC::ci::ci.roc.test?

Describe alternatives you've considered
No other suggestions...access to the estimates described above in any way would be great.

Additional context
If the described solution above is sufficient, would like a chance to submit a pull request. Please let me know if that is acceptable.

Implementation status

For quick reference, here is the status of this request:

• DeLong
  • paired (add confidence interval output to paired DeLong test #95)
  • unpaired
• Bootstrap
• Venkatraman

[xrobin]

What would you like to return exactly? From what I understand, you want to calculate the CI of the estimates, so changing:

> htest <- roc.test(roc1, roc2)
> htest$estimate
# AUC of roc1 AUC of roc2 
# 0.7313686   0.8236789 

Into:

# AUC of roc1 (95% CI)  AUC of roc2 (95% CI)
# 0.7313686 (0.7 - 0.8)  0.8236789 (0.8 - 0.9)

How is that different from running ci.auc(roc1) and ci.auc(roc2)?

> ci.auc(roc1)
95% CI: 0.6301-0.8326 (DeLong)
> ci.auc(roc2)
95% CI: 0.7485-0.8988 (DeLong)

Can you show what you'd like to see a bit more specifically? (Doesn't need to be the exact values).

In addition is this something you've seen implemented in a different test or in a different package? Do you have a reference that describes it, or a similar approach for a different test?

[benneely]

I was thinking along the lines of:

> htest <- roc.test(roc1, roc2)
> htest$estimate
[1] -0.0923103

Would like the CI of this estimate. This would involve altering the interface for the delong functions to get SEs derived in those functions. It looks like the appropriate way to get this information might be of the form?:

ci(htest, of="roc.test")

I haven't seen this implemented elsewhere.

[xrobin]

Now you're talking about the statistic, not the estimate, right?

I guess that should be doable pretty easily DeLong describes the expression for a confidence interval of L\theta' on page 841 of their 1988 paper (unnumbered equation, the 2nd on the page), and most of the code should be in delong.R already (the delong.unpaired.test and delong.paired.test functions specifically).

I wouldn't call it ci.roc.test as you it's not really assessing the confidence interval of the hypothesis test itself, but it could be for instance ci.delong or something similar? Also do you think pROC should expose any other related function to the end user that might be useful?

I'll be happy with a pull request! You can branch off directly from master.
The DeLong part is pretty well tested so feel free to re-use/refactor whatever you need there as much as you like. Mind the tests in test/testthat, and the doc that's still old-style Rd files in man/ (no roxygen at the time most of the code was written).

[benneely]

Great!

...but it could be for instance ci.delong or something similar?
Sounds good - I will use this naming scheme.

...do you think pROC should expose any other related function...
In my cursory review, the only item that I wished I had access to was SE. Will plan to touch base and confirm about exposing any additional objects.

Thanks for the helpful information about documentation and testing. Will give this a shot and try to have a PR in ~week.

[xrobin]

Any follow-up on this feature? Did you have a chance to give it a try?

Of note, a possible use case was posted to stack exchange (Cross Validated): https://stats.stackexchange.com/q/461992/36682

[rickeycarter]

I have been looking around in R for a similar need -- generate the estimate and 95% CI of the difference. Something like this as output would be lovely:

roc.test(test1, test2)

Test 1: {auc1} 95% CI: ({lcl1} to {ucl1})
Test 2: {auc2} 95% CI: ({lcl2} to {ucl2})
Difference: {auc1 - auc2} 95% CI: ({lcl_delta} to {ucl_delta})

[wzbillings]

Are there any references for the CI for the unpaired test? I've been looking around a bit and I can't seem to find any literature that derives an equivalent CI in the unpaired case.

[d-morrison]

for the unpaired CI, would it be reasonable to use a Gaussian approximation for the difference (pooling the SEs)?