My contact lens fell out onto a relatively clean surface. I picked it up and put it back into my eye without washing my hands or the contact. There were no visible particles on the contact and the contact is sitting comfortably in my eye with no signs of redness or irritation. It is ok to wear this for a couple of hours right?
Much like you probably won't catch a cold every time someone sneezes on you, you also most likely won't get an eye infection every time you neglect contact lens hygiene. We all did it at some point, I'm sure.
However... just because you can't see the wildlife on your contacts and hands, doesn't mean it's not there. It is. Little buggers like this one https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4330640/
Bottom line: you'll probably be fine. Probably, not certainly.