A toric lens is the type that is used to treat astigmatism, not an aspheric lens. An aspheric lens is primarily designed to correct for the spherical aberration of the contact lens itself, the spherical aberration of the combination of the cornea and lens of the eye, or both.
A toric lens is shaped like a slice of the side of a torus. It has a different index of refraction along its horizontal axis than it does along its vertical axis, with a smooth gradient of the refractive power in the quadrants between those axes. Astigmatism is the condition where the corneal surface, the lens of the eye, or both have the same kind of difference in refractive power along the horizontal and vertical axes of the eye. The toric lens's corrective axes must line up correctly with the astigmatic axes of the eye. A common method of accomplishing this is by making the lens heavier near one part of the edge so that the heavy edge self-rotates to the bottom.
An aspheric lens, on the other hand, has a gradually changing refractive index moving radially from the edge of the lens to the center. It will either be "flatter" or "steeper" than a purely spherical lens, but the refractive index is constant moving along any circumferential path at a constant radius from the center of the lens. The magnitude of the lens's deviation from being purely spherical is known as its spherical aberration. An aspheric corrective lens is used to correct for the inherent spherical aberration that a spherically-surfaced contact lens introduces, which is directly proportional to its sphere power, and ideally also the spherical aberration of the eye itself.
Simply by the properties of optics, the effect of using either of these types of lenses if one does not have the corresponding visual defect the lens was designed to correct would be to introduce some degree of distortion into one's vision. This could only negatively affect visual acuity, though how severely would depend on the particular lens's strength. I could not find any references as to whether there are any long-term harmful effects of intentionally distorting a person's vision or visual acuity since the subject of all the references I checked was trying to achieve the opposite effect. The only mechanism I can postulate that could potentially cause damage to the eye would be by the long-term eye strain induced by the intentional distortion of a person's vision.
Here are two helpful references I found in further elaborating the above information:
http://www.clspectrum.com/issues/2005/march-2005/aspheric-contact-lenses-fact-and-fiction
https://coopervision.com/about-contacts/toric-contact-lenses