It's harmful, because what passes for the rest of the so-called "well-balanced diet" isn't actually all that healthy either, it will also contain a fair amount of sugar, and saturated fats and also unsaturated fats which are healthier than saturated fats but not healthy in an absolute sense. So, what passes for a healthy, well-balanced diet is a diet that isn't too bad, but certainly not the most healthy diet you can imagine. This should be considered to be a compromise between what is extremely healthy and what the general public can be convinced to stick to.
If you then further compromise on this by getting about 10% more of your calories from sugars and fats, of which a large fraction will be saturated fats, you're going to do yet more damage to your body. To get to a healthy diet, you must get rid of almost all the salt, sugar and fats in the diet, even so-called "healthy oils" like olive oil as Dr. Klaper points out here.