There seems to be a link (as one would expect) between areas with a higher concentration of unvaccinated individuals and disease outbreaks (see this article and the source data map).
The recent (2015) measles outbreaks in California are a good example: at least 70-80% of those infected were unvaccinated (including a number of newborns too young to be vaccinated).
State officials say that 28 were not vaccinated at all, one was partially vaccinated and five were fully vaccinated. Six of the unvaccinated were babies, too young to be vaccinated.
Simply put, unvaccinated individuals are more likely to be carrying a vaccine-preventable disease compared to the rest of the population. If your newborn is not vaccinated yet he or she is at risk. In the case of measles or whooping cough, an infection could turn deadly, as can many others.
I'm not an expert in pathology, nor am I a parent, but I would at the very least not let these individuals hold or come near your newborn, and would preferably keep my newborn completely away from them until a later time.