# Does this paper report miscalculated metrics?

During a literature search I stumbled upon this paper, which evaluated the diagnostic accuracy of ultrasonography for the detection of gallstones (compared to the gold standard cholecystectomy).

Their data looks as follows:

• TP: 1549
• FP: 274
• FN: 0
• TN: 46

According to wikipedia, sensitivity is calculated as `TP / (TP + FN)` and specificity as `TN / (TN + FP)`. So if we plug in the numbers, we get:

• Sensitivity = 1549 / (1549 + 0) = 1
• Specificity = 46 / (46 + 274) = 0.14375

Yet in the abstract they state that they found a

sensitivity of 0.85 and a specificity of 1 for ultrasound in the identification of gallstones

Is this just wrong or am I missing something? Would be glad if somebody could help!

As the other answer points out, the authors here have clearly calculated the positive and negative predictive values (PPV and NPV) rather than sensitivity/specificity. They report these clearly in their results section, but everywhere else use the words sensitivity and specificity.

There is a reason that it might not make a lot of sense to consider sensitivity and specificity in this study. Those measures are meant to be real-world measures that are most meaningful when you consider them in light of actual prevalence.

However, look at prevalence here: only 320 people didn't have gallstones, and 1549 did! This is kind of ridiculous - about 80% of people have gallstones! How can this be? Well, the other problem with their study is that they're relying on a gold standard metric that simply isn't available for most people without gallstones.

Why? Their sample is people underwent laparoscopic cholecystectomy - they had their gallbladders removed. Specificity is always going to be terrible in this sort of selected group, because you're missing all the people who didn't have their gallbladders removed, most of whom likely would have been negative on the ultrasound.

My guess is that they calculated specificity and sensitivity to start, got numbers that didn't agree with the literature, and went to a statistician or colleague for an explanation. Someone told them that PPV and NPV are likely better measures for this particular situation, and they sloppily edited the paper without changing the other sections, perhaps because they didn't realize (or it was explained incorrectly to them) that these are actually different than sensitivity/specificity.

I'll note that based on their affiliations, the authors seem likely to be physicians. They probably have minimal training in research and statistics; it's possible one or both were even medical students when they wrote the paper.

• Thanks, that makes sense! I'm still curious how this passed peer review… In my understanding, it is PPV and NPV that are sensitive to the prevalence in a population, whereas sensitivity and specificity should not be. However, your point with the sampling error (cholecystectomy) makes a lot of sense. They even excluded damaged gallbladder specimens from the evaluation, otherwise the specificity might be even lower (because of potentially lost stones).
– asdf
Jul 25, 2022 at 17:01
• @asdf The problem here is that it isn't population prevalence but the prevalence in the gallbladder-removal group. Sensitivity/specificity should stay consistent in an unconditioned group, but they can't be estimated from a conditioned one. Of course the PPV/NPV estimates are also impacted but it's more easy to change the interpretation of those in the context. But yes, peer review is highly imperfect and the reviewers may not be experts in statistics, either. I'd also say that review in the medical world tends to be much poorer for these sorts of retrospective analyses versus designed trials. Jul 25, 2022 at 17:06
• Ok, that helped a lot! I'll accept your answer, thank you!
– asdf
Jul 25, 2022 at 17:17

It seems that they meant to say PPV(positive predictive value) and NPV instead of sensitivity and specificity.

This provided a positive predictive value (PPV) of 0.85 and a negative predictive value (NPV) of 1.

In the results section, you can see that nowhere they mentioned about sensitivity and specificity but only PPV and NPV.

Is this just wrong or am I missing something? Would be glad if somebody could help!

You are not missing anything, their calculation is not sensitivity and specificity.

• Thank you for your answer! But then their conclusion does not make sense either, since they state that `Meta-analysis of studies between 1966–1992 assessing the diagnosis of biliary disease produced figures of the adjusted sensitivity of ultrasound for the diagnosis of gallstones at 0.84 and a specificity of 0.99. These figures are similar to the figures that we collected nearly 20 years later.` The meta analysis they cite here seems to report clearly different results then. A specificity of 14% is certainly not similar to 99%...
– asdf
Jul 25, 2022 at 12:37
• Yes, the study seems to have a lot of holes as mentioned in @Bryan's answer. I think they did this study just to get a postgraduate degree (MD) as it is a requirement in the UK that one should submit an original research. Jul 25, 2022 at 15:33