According to the NHS, on average, every 15 cigarettes will cause one genetic mutation.
Usually that mutation will do nothing sinister, but every one has the potential to develop into cancer.
Mutations aside, lots of nasty things happen immediately when you smoke. The filter doesn't catch much of the filth, which very rapidly enters your bloodstream. The body does many things to deal with toxins, and has an emergency last resort technique for dealing with scenarios where a large volume of toxins enters the system in one go, and can't be removed fast enough. The toxins are locked up in fat cells around the liver, to effectively take them out of circulation. If you lead a very healthy lifestyle, and maintain an ideal weight, then over time, those stored toxins will be released gradually and disposed of via natural detoxification processes. If you don't have a calorie deficit though, then your body never needs to empty its fat reserves, so that toxic fit just stays around your liver indefinitely.
All that said, I doubt that 20 cigarettes every few months will do noticeable harm, except for making you stink. Stress can cause harm, if not physically then certainly mentally. Perhaps a bigger risk though is the thought habit you are developing of justifying smoking by blaming stress. As an ex-smoker, I know this cycle too well. There is a risk that you might gradually increase the frequency of smoking, each time blaming stressful situations.
Perhaps instead of wondering how much harm your smoking is doing, perhaps it would be better to tackle the underlying stress, or find an alternative stress relief.