Eating chalk is a type of Pica. It is characterized by an appetite for substances that are largely non-nutritive, such as paper, clay, drywall or paint, metal, chalk, soil, glass, or sand which is more common with women and children.
Low blood-hemoglobin levels, a sign of anemia, are common, because the ingestion of chalk inhibits absorption of iron from foods.Those who eat chalk at the expense of healthier foods often develop malnutrition.Source.
Due to inhibition of absorption of iron from food, it results in deficiency of iron in our body.Low blood iron levels limit our body's ability to produce hemoglobin, the oxygen-carrying protein in red blood cells, which might be possible cause of causing anemia.
So, yeah your doctor is right, consumption of chalk can really cause severe low Hb level.
Pica and refractory iron deficiency anaemia: a case report