My understanding is that psychological stress is known to have short term (transient) effects on raising blood pressure, but there's less evidence that psychological stress is a risk factor for hypertension (high blood pressure for a long period of time).
Here's a meta-study of other studies:
Acute stress promotes transient elevation of blood pressure, but there
is no consistent evidence that this effect results in hypertension.
http://www.nature.com/jhh/journal/v23/n1/abs/jhh200874a.html
As @JohnP says, doctor's office blood pressure is known to be a suboptimal test of hypertension. Its a single sample, and also it may show up higher (as in the case of the White Coat Effect), or lower (as in the case of Masked Hypertension) than normal. Better determinations of hypertension can made by either wearing an "ambulatory blood pressure" cuff for 24h, or by making many readings with a home BP monitor (in which case your must carefully follow the instructions, be sitting, rested and keep you arm at the correct height). Bear in mind however, that readings from ambulatory and home BP devices are not directly comparable to the same BP values seems in your doctor's office. Different values correspond to the same risk (lower BP values are generally used to determine hypertension when the readings are made out of the doctor's office).

In the above image, "home" is the sitting self test, 24h is the "ambulatory" (while active) average, and nighttime and daytime are the averages of the ambulatory monitor when when asleep/awake. Its important to know there's generally (very!) large difference in risk between the same home and office readings.
The fact that your BP returns to truly excellent levels may a good indicator in that your body is reacting appropriately to demands. That said, if you have prolonged high BP (due to stress or otherwise), you should be concerned.