This all falls to a matter of degree - most importantly mechanism of injury. I'll explain the physics in allegorical terms first and then explain as it relates to medicine.
The mechanism refers to the amount of force experienced in the injury process which allows us to predict injury patterns.
For example: A motor vehicle collision at highway speed has a very high mechanism of injury as there are a lot of forces involved (mainly kinetic energy). Stubbing your toe has a very low mechanism as the force involved is exponentially less (although it still hurts and can still cause a fracture).
In the case of kicking or punching specifically; I think we must consider the physics. The forces involved are not from an outside source and occur as a result of your own body mechanics.
Your body is designed to withstand the forces of movement, even in extreme settings such as: fighting a predator or attempting to flee from one.
Generally when you punch something you do so by first loading the hips with potential energy and then converting it to kinetic energy which is then transferred into the object being hit. Some of it does get reflected back but most of that is absorbed and (with good biomechanics) transferred into the floor through the long bones - just like a bicycle spoke. Our flexible neck also allows our body to move independently of the head to absorb/redirect kinetic energy away from our vital organs.
In the case of the football player, however; things are different. Outside forces are at work which our bodies are not necessarily equipped for. A 260 lb football player in full equipment who is running head-on toward you, bracing right before the hit, and doing so with the intention of transferring that energy to you is more than your body can effectively compensate. This is where injury occurs and this will have a negative effect.
Now to dive into the medical terminology a bit:
Brain injury is caused by 4 main mechanisms:
- Sudden Impact
- Rapid Acceleration/Deceleration
- Penetration Blast Injury
They are also classed in 3 different categories:
- Diffuse Axonal Injury
- Focal Contusions
- Hematomas (bleeding, in or around the brain)
Milder impacts usually are limited to Diffuse Axonal injuries. The white matter is injured diffusely at the cellular level, damaging the axon terminals of many neurons throughout the brain. These occur primarily from sheering forces related to sudden acceleration/deceleration.
Even though you may experience this in something like gymnastics, it is generally minor in nature and does not cause enough damage to actually bruise the brain. If you fall while doing gymnastics and hit your head on something directly ( like being tackled helmet to helmet) it could bruise the brain by causing a coux and/or contre-coux injury.
Our brains are insulated with a layer of fluid between it and the skull - but a strong and sudden impact can cause the brain to hit the skull.
When the head moves too quickly for the brains resting inertia to smoothly go with it the brain hits the skull's interior wall. Much the same way you may hit your face on the dash of a car if someone slams the brakes and you aren't ready and wearing a seat-belt.
On top of this there is also another impact of the brain against the opposite side of the skull as the brain by itself cannot dissipate that recoiled energy from the first impact. The result is said to be a coux-contrecoux impact/injury. This results in a bruising of the brain at the point of initial impact, as well as the point opposite.
If the mechanism is high enough it can even cause a hematoma, which is basically a brain bleed and is a medical emergency. If left unchecked these can often be fatal. As pressure in the head increases due to blood accumulation in the closed intracranial cavity the brain will eventually begin to herniate through the base of the skull.
I hope that answers your question. If you want more information about intracranial bleeds I suggest researching "Cushing's Reflex/Triad".
Here's a link to a video explaining it further as well as a reference for more detailed information : https://www.youtube.com/watch?v=U9x_s6c0EGQ