This depends, unfortunately. And it does depend a bit on budget as well.
Electric kettles used for boiling water can leach certain substances.
For plastic the main concern currently is BPA leaching.
What is BPA and Why Should Your Electric Kettle be Plastic Free?
Do the plastic in kettles leach the toxin BPA?
Why Electric Kettles Should Be Plastic Free
There are two sub-questions that must be addressed here.
Is BPA used in can linings, and can it leach into food?
This is definitely true. The National Institute of Environmental Health Sciences (part of the NIH), for example, says that
Polycarbonate plastics have many applications including use in some food and drink packaging, e.g., water and infant ...
If talking about the container itself and the chemicals it might leave on the product being consumed then these fears are mostly related to the fact that BPA (bisphenol A) is present in poly-carbonate bottles and on the adverse effects this has on the brains and development on reproductive organs of mice.
However, when compared and related to humans, the ...
Prepared food is typically good for 3-4 days in the refrigerator
food safety guidelines
Frozen is is even longer.
It would be cooled (not frozen) in a normal refrigerator.
Healthy depends on the food. Home prepared food can be healthier than prepared food (especially fast food) that can have a lot of fat and salt.
I agree with the answers on cleaning and the type of bottle/cork/lid.
Another thing you should consider is that residue from your saliva may be what smells bad. Try leaving the freshly-cleaned bottle filled with your usual water out for a day without drinking from it and see if there is any odor, in which case it might be your water source. Then try ...
Plastic bottles work just fine. When buying sparkling water (I'm German), they come in plastic bottles and never get this taste of old shoe soles, and I haven't had any problems with bacteria either.
In your case, the cork could be a problem, as was pointed out before. I personally have made the experience that water in steel bottles always tastes a bit ...