No, gynecomastia is a hormonal condition in which mammary tissue develops and enlarges in a male. Pseudogynecomastia is the word for what you're describing.
How do you know what you have is only pseudogynecomastia? Have you consulted a doctor and know for sure? If not, you need to stop looking for solutions and look for the cause first. You can't solve it if ...
Seconding Carey, it is most likely (not impossible) that your "saggy" chest is due to Pseudogynecomastia, which is caused in your case by losing weight too fast.
If male breast enlargement is caused by glandular proliferation, it is
defined as gynecomastia. If it is caused by increased fat deposition,
it is defined as pseudogynecomastia
Erol S, ...