Update: April 2 news is that the WHO has now commissioned a panel to study the problem further:
Should more of us wear face masks to help slow the spread of coronavirus?
This question is to be assessed by a panel of advisers to the World Health Organization (WHO).
The group will weigh up research on whether the virus can be projected further than previously thought; a study in the US suggests coughs can reach 6m and sneezes up to 8m.
The panel's chair, Prof David Heymann, told BBC News that the new research may lead to a shift in advice about masks.
The former director at the WHO explained: "The WHO is opening up its discussion again looking at the new evidence to see whether or not there should be a change in the way it's recommending masks should be used."
Original answer below:
The effectiveness of masks in the general population, as opposed to use by HCW (health-case workers) has not been studied a lot. Some such studies involved "home use" which may or may not be equivalent to "going shopping" use, so we don't have a lot of clear data on how people might be using masks in the present pandemic.
But some facts that were reported...
In the 1919 influenza pandemic, masks were available and were dispensed to populations, but they had no impact on the epidemic curve.
There was no clear idea how to design masks at that point, a "germ-proof" surgical mask was only available from the 1930s onward.
And some "home use" RCTs; MacIntyre et al. (2009)
We conducted a prospective cluster-randomized trial comparing surgical masks, non–fit-tested P2 masks, and no masks in prevention of influenza-like illness (ILI) in households. [...] Adherence to mask use was associated with a significantly reduced risk of ILI-associated infection. We concluded that household use of masks is associated with low adherence and is ineffective in controlling seasonal ILI. If adherence were greater, mask use might reduce transmission during a severe influenza pandemic.
Cowling et al. (2009)
INTERVENTION: Lifestyle education (control) (134 households), hand hygiene (136 households), or surgical facemasks plus hand hygiene (137 households) for all household members.
RESULTS: Sixty (8%) contacts in the 259 households had RT-PCR-confirmed influenza virus infection in the 7 days after intervention. Hand hygiene with or without facemasks seemed to reduce influenza transmission, but the differences compared with the control group were not significant. In 154 households in which interventions were implemented within 36 hours of symptom onset in the index patient, transmission of RT-PCR-confirmed infection seemed reduced, an effect attributable to fewer infections among participants using facemasks plus hand hygiene [...]. Adherence to interventions varied.
Basically, masks can work for the "average Joe" but note that some effect of hand hygiene was observed even without masks, but not vice-versa.
Here's the full summary table from the review of MacIntyre and Chughtai (2015) on household/community settings studies. (Paper found by Vladimir F.)
As you can see, most report no significant differences for masks alone. The review says that a meta-analysis is not possible due to varying study design(s). It also says the studies may have been underpowered. They discuss the studies in a bit more detail (including the two I've already mentioned above). Since the ones inside a household are pretty similar to each other, let me quote just what they say on two studies on a slightly different setting:
Two RCTs in university residence halls in the United
States over two influenza seasons randomised well students
into medical masks plus hand hygiene, medical
masks alone, or control. Influenza-like illness and
laboratory confirmed influenza were not significantly
reduced after either intervention, although during the first
four to six weeks, influenza-like illness was significantly
lower in the medical masks plus hand hygiene arm in
both trials (P<0.05). This suggests that hand hygiene
might have been the major contributor to protection.
(Those are the two studies by Aiello et al. [2010, 2012] in that big table.)
The review also says:
The routine use of facemasks is not recommended by WHO,
the CDC, or the ECDC in the community setting. However,
[citing the same guidelines] the use of facemasks is recommended in crowded settings
(such as public transport) and for those at high risk
(older people, pregnant women, and those with a medical
condition) during an outbreak or pandemic.
However... looking at the actual sources cited 2009 CDC page and 2006 WHO paper... what they actually say (the CDC in particular) is that both criteria (crowded setting and vulnerable person) have to be met simultaneously for the mask to be recommended.