1

I understand how to evaluate therapeutic studies: basically you compare the results of two groups that were chosen randomly. One group gets the new drug, the other group gets the placebo (or the drug against which the new drug should be tested). Neither the physicians nor the patients know in which group they are (double-blind testing). You can clearly see the results and test the statistical significance (e.g. with a t-test, depending on the study design).

I have a problem understanding diagnostic studies. My problem lies in the fact that you do not know "the truth", i.e. whether a patient really has the condition which should be diagnosed. Therefore you use the so-called "gold standard", i.e. the best diagnostic method so far. There is no problem when the new diagnostic method is worse than the gold standard. Yet I think there might be a problem when the new diagnostic method is better than the gold standard! Why? Because the results will be different which would be interpreted as inferior.

Let me give you an example: Let's assume that a patient really has some form of cancer, yet the gold standard is not able to detect it. The new method is able to detect it and therefore both tests disagree. Because the gold standard is taken as the reference the new method would get a minus point here... although it was right in the first place!

My questions
Is my thinking correct? If yes, how do handle this problem in practice? Is there a common term for this kind of problem? If no, where lies my misunderstanding?

Edit
Perhaps another (more extreme) example is in order: Let's say the gold standard has an accuracy of 50%, i.e. the toss of a coin. If you had a new method with an accuracy of 100% (so the results of this method and "the truth" are the same) and tested it against the gold standard this new method would get an accuracy of only 50% (i.e. the accuracy of the gold standard against "the truth").

Improve this question
7
  • This question has a perfectly good answer here: en.wikipedia.org/wiki/Gold_standard_(test)
    – JMP
    Commented Oct 6, 2019 at 7:38
  • @JMP: So, it really is a problem... and the solution is "calibration"? But how do you "calibrate"?
    – vonjd
    Commented Oct 6, 2019 at 7:42
  • Are you asking how to determine the performance of a competitor to the gold standard? That would make a much better question.
    – JMP
    Commented Oct 6, 2019 at 7:46
  • @JMP: Yes, this is what I am asking: "how to prove that new method is better than gold standard?", the "new method" is the "competitor". In the Wikipedia article I only see that the problem is acknowledged but I don't find a solution on how to address it.
    – vonjd
    Commented Oct 6, 2019 at 7:49
  • 1
    The truth comes out eventually. The person goes on to show symptoms, or the biopsy reveals the presence or absence of a mass, or the CT or MRI shows there is a swelling/thickening/narrowing or whatever it was you were trying to diagnose.
    – Kate Gregory
    Commented Oct 6, 2019 at 13:16

2 Answers 2

Reset to default
1

The accuracy of a new diagnostic test can be determined by comparing the test results with the results of a biopsy (taking a piece of tissue), which is done at some point after the test, and which should reliably confirm/exclude the pathology in question.

For example, they tested the accuracy of "multi-detector CT" in differentiating between gallbladder inflammation and cancer. They compared what the CT has shown and what the biopsy after gallbladder removal has shown. So, they did not compare the results of the new test with the "gold standard test" but with the cases of cancer they proved later by biopsy.

They use the mentioned process not only to evaluate the accuracy of new tests but also to evaluate the results of the existing tests. For example, they try to determine if a certain "shadow" in the ultrasound image speaks for gallstones or cancer (which they can see after gallbladder removal).

Improve this answer
2
  • Thank you, Jan. Although it seems that a biopsy is pretty reliable I would think that it is also not the same as the "ground truth". An example where this could become a problem is evaluating the accuracy of interpreting the results of the biopsy itself, e.g. in a study "AI vs. pathologists"...
    – vonjd
    Commented Oct 7, 2019 at 15:37
  • 1
    @vojnd, today, in many cases, biopsy results are considered a proof of the presence or absence of pathology. It is true that biopsy can miss something, but this is what you can do. Sometimes, the pathology needs to develop further to be detectable and, sometimes they can detect it only at autopsy - and they also use these data to determine the accuracy of various tests. You do not need to think if these approaches are not 100% accurate that they are not useful.
    – Jan
    Commented Oct 7, 2019 at 15:40
0

Wikipedia says:

Before widespread acceptance of any new test, the former test retains its status as the "gold standard".

which means that a competitor must perform better than the current gold standard over repeated and diverse tests.

Single isolated cases like the examples you have given do contribute to the overall data, but are also to be treated as possible 'needles in haystacks'. Numerous tests are required to eliminate anomalies, and a variety of statistical methods can be used to determine if anomalies can be viewed as exceptions to the rule.

In summary, no single test will suffice. It takes a multitude of tests over a variety of patients and conditions for a procedure to be validated.

Improve this answer
1
  • Thank you, I really do appreciate your answer! Still, even "a multitude of tests over a variety of patients and conditions" cannot solve the systematic problem I address in my question. No test where you use the current gold standard as reference can prove that your new test is better than the gold standard. You would have to test it against the "ground truth" which can be super problematic and time-consuming...
    – vonjd
    Commented Oct 6, 2019 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.