Originally the question was
How are Sensitivity/Specificity determined for the first test available for a disease? but I narrowed it down a bit since it seemed to be too broad.
Given that measuring Sensitivity and Specificity requires that you know in advance which people are positive or negative for some disease what's the general process if you develop a new test for a new disease where there's no existing test for the disease making it hard to actually reason about who has the disease and who doesn't. Similarily, if you have an existing test and develop a new test that is supposed to be more accurate, how are you going to tell who has the disease and who not because using the existing test to quantify that introduces "unkown" errors.
Similarily, how is this done for diseases that don't even have biomarkers available at that point. I.e. mental illnesses like depression. The way they are defined is by a set of criteria that define the illness and then people develop questionaires to screen for these mental illnesses but since the only way of knowing whether somebody actually has depression is to apply the criteria from the definition those criteria are actually what defines the test in the sense that hose criteria determine whether somebody has the disease, or not. In essence, doesn't this mean that you invent criteria based upon 'nothing' and then create a questionnaire for it but since the definition is actually arbitrary how does this gain you new insight or new information? In some sense, I could define 'hoarding' as well... 'excessively collecting/hoarding objects...' and then devise a questionaire that consisting of two questions 'do you hoard books' and 'do you hoard empty bottles' and if 80% of hoarders do one of these things I will have 80% sensitivity and 100% specificity but I might as well just use the criteria from the definition and do much better than that questionnaire.
You can't really make the argument that the criteria are set so that it fits people having the disease well... because the criteria is what defines the disease in the first place contrary to disease that are diagnosed based on an underlying known pathology and tests try to detect that pathology (or side-effects of that pathology) and the accuracy of the test depends on how good in can detect that pathology.
I guess the way I'd do it is I'd just measure a lot of stuff and then see if I can find any significant correlation. Like I have 100 patients with similar symptoms and I check their blood and see that 80 of those patients have more copper in their blood than a healthy control group causing me to think that copper is involved in the pathology then I'd investigate that further and then I notice that their bodies don't process copper as they should and this sometimes causes in 90% of the people with that pathology increased copper blood levels, in 10% we don't see that and the other 10% of my patients turned out to have something that produces the same symptoms but the pathology is something completely different. So I devise a test around that that' measures elevated copper levels which 90% people (with that pathology) have but with some errors and due a cut-off level I have to put somewhere let's say I get about 88% sensitivity and 96% of specificity (because a few people have idiopathic elevated copper levels or something that produces a higher copper level). But I obviously can't really do the same for illnesses where I don't know the pathology because there's zero way of telling who has which pathology. Like depression. If I define depression as 'a.) has low mood on most of the days for at least 6 months' or similar and this is both the test AND the definition of the disorder I obviously get 100% sensitivity, but that's because the test and the definition is the exact same thing and there's no way of determining the specificity because everything that matches my criteria is by definition having this illness I just defined meaning it will match absolutely everything that provokes these symptoms regardless of the pathology so I can't really measure specificity unless I neglect the existence of other diseases producing the same symptoms, in which case I'd get both 100% sensitivity and 100% specificity but this is obviously 'cheating' OR if I just define it as co-morbid to some other pathology in every case but that's basically also just a neat 'cheat'? I could then add a little extra criteria 'b.) not due to some other disorder' but this at least scientifically looks like another 'cheat' because now you don't have a test for anything anymore to begin with. You have a definition, but no test. You can't use a questionnaire anymore.