If you are not pleased with what the doctor said or you do not trust him, obtain your medical documentation about your case and ask for the second opinion.
Another doctor, in order to be able to give you a proper advice, will need to know the exact type and size of the polyp, was it the only polyp, the first polyp, etc.
Here is an example of guidelines for the screening after a polyp removal:
Colorectal Cancer Screening and Surveillance (American Family Physician, 2015)
Patients with one or two tubular adenomas that are smaller than 10 mm
should have a repeat colonoscopy in five to 10 years. Repeat
colonoscopy at five years is recommended for patients with
nondysplastic serrated polyps that are smaller than 10 mm. Patients
with three to 10 adenomas found during a single colonoscopy, an
adenoma or serrated polyp that is 10 mm or larger, an adenoma with
villous features or high-grade dysplasia, a sessile serrated polyp
with cytologic dysplasia, or a traditional serrated adenoma are at
increased risk of developing advanced neoplasia during surveillance
and should have a repeat colonoscopy in three years.
I don't want to give any suggestion or hint other than: you may want to ask for a second opinion.
When a doctor removes a small polyp from the colon, it is usually adenoma - this is a benign, noncancerous tumor that does not cause any harm. But if you leave such adenoma in the colon intact, it can often develop into a cancer.
Also, once one adenoma is found in your colon, you are at increased risk that another adenoma will appear later and if you give it enough time, it can develop into cancer. I don't know the exact statistics right now.
Stool tests for colorectal cancer are briefly explained on CDC.gov:
"FIT" test is basically just more specific test than it is the more known guiac fecal occult blood test (gFOBT).
The "FIT-DNA" test (also referred to as the stool DNA test) combines the FIT with a test that detects altered DNA in the stool.