This is a great question, an area of active debate, and a personal interest of mine. I actually just did a presentation on this, so this answer will probably have more information in it than you need, but I'll add it for completeness.
Fever as an Adaptive Response
The first important thing to know about fever is that it is something your body initiates when you get certain types of infections. It is not a direct consequence of the bacteria or viruses themselves. Instead, white blood cells release a chemical messenger called prostaglandin E2. This travels to the brain, where it causes the body to start ramping up its set point for temperature-- essentially turning up the thermostat.
This is important because it seems to indicate an evolved response-- perhaps there is an evolutionary advantage to a fever in the setting of infection. Many bacteria are very temperature sensitive, so this idea makes sense.
Many papers have been published on the effect of fever on infection-- here are just a few:
In critically ill patients (my area of interest), there have been several interesting studies. In this randomized controlled trial (RCT) from 2005, they compared mortality in ICU patients where fevers were controlled with acetaminophen/ paracetamol and where fevers were not controlled. They actually had to stop the trial early because death rates in the group getting acetaminophen/ paracetamol were much higher.
This prospective study (not randomized) from 2012 looked at over 1400 critically ill patients, and found that treating fever doubled the mortality rate.
This meta-analysis, which included 11 studies, found a "trend towards higher mortality for aggressive treatment" of fever, though their p-value (how likely it is their findings are due to chance) was 0.09, which is higher than the 0.05 that is traditionally used as the cutoff for statistical significance.
The most high-powered meta-analysis done to date, which included 16 studies and a total of about 20,000 critically ill patients, found less dramatic results-- essentially concluding that it makes no difference if you treat a fever or not.
Most of the studies done here are in critically ill patients, with mortality as the final endpoint. You can make a case that in those patients, you may be better off not treating fever, since you want to keep them alive.
The vast majority of fevers are due to respiratory viral illnesses. These have low mortality rates, and so rather than trying to lower mortality, you're trying to minimize the symptoms. And it sucks to have a fever. So for most people who do not need to get admitted to the hospital, treating a fever is a tradeoff between potentially getting better faster (an UNPROVEN benefit, as far as I could find), and not having a fever.
If a person is willing to sweat it out with the hope that the disease will run its course faster, that's perfectly reasonable. And if they are ok with the disease taking a little longer to clear from the body as long as they aren't sweating, shaking, and burning up, that makes a lot of sense too.
I should also note that there are times when a fever IS extremely dangerous-- if it's over about 106 it can lead to brain damage. All of the studies in humans treated fever if it went above about 104 degrees.
TL;DR: You are onto something! There is a lot of data out there that supports the hypothesis that treating fever is bad for you (or at least not any different from doing nothing).