Looking at the aforementioned Wikipedia page, which indeed confirms that (historically) variolation was widely accepted in the US at some point, I can't seem to find any scientific analyses of its effectiveness listed there. So, have such studies been conducted, either in terms of analyzing the historical data through a (counterfactual) epidemiological lens (iffy I suppose), or at least in contemporary animal studies?
(Obviously, the practice was abandoned once a vaccine was developed.)
Variolation served as a natural precursor to the discovery of vaccination. The major differences between the two were that in vaccination, material from cowpox, an animal disease, was used, but particularly that it was safe to those vaccinated and was not transmitted to their contacts. Vaccination offered the public a less-harmful method of preventing smallpox. Vaccination would revolutionize the control of smallpox, leading to its eventual eradication.
I see there's a University of Colorado page that says:
During the smallpox epidemic that swept across North America from 1775 to 1782, Revolutionary War soldiers took an unusual approach to protecting themselves from the virus known as Variola major. In a process known as variolation (a.k.a. inoculation), they took virus-loaded material from an infected person’s smallpox pustule, carved an incision into the flesh of a healthy solder, and rubbed it in.
Recipients of variolation invariably got the disease, so were quarantined. About 5% died. But most got a mild version of the smallpox disease.
“There is no question that it worked,” says Fenn. “Assuming you lived through it, you would garner immunity and go about the world without worrying about smallpox.” [...]
Fast forward to today, and variolation has come full circle, as researchers explore the idea of using “convalescent plasma” (survivor blood believed to contain antibodies to COVID-19) as a treatment.
So apparently some (historical, at least) studies might have been conducted, but they're not clearly listed there. (I think the whole page is based on an interview with "CU Boulder history Professor Elizabeth Fenn, a Pulitizer Prize winning writer, scholar of epidemics and author of Pox Americana: The Great Smallpox Epidemic of 1775-82.")