This depends a little on whether the inhaler is a metered-dose inhaler (MDI - aerosol) or dry powder inhaler (DPI). DPI inhalers require adequate inspiratory flow rates.
Metered dose inhalers
Propellants in MDIs typically make up more than 99% of the delivered dose and are usually hydrofluoroalkanes (HFA). These a gaseous and are usually exhaled. I cannot find any data on the extent to which (if at all) that HFAs may be absorbed.
Dry powder inhalers
Dry powder inhalers (an example) mainly use lactose (a milk sugar, usually as alpha lactose monohydrate) to form the bulk of the powder because it is safe and easily available. This is usually absorbed by the mucosal lining of the lung and does not accumulate over time. It can also be deposited in the mouth and throat and ingested into the gastrointestinal tract. From here it can be absorbed to some extent (though lactose undergoes less active absorption in the gut than other sugars) or egested.
Drug absorption by inhalation
In fact, traditional aerosol devices can only deliver about 10 to 20% of their loaded doses into the lungs - source. A lot of this rest ends up in the gastrointestinal tract. Drug particles less than 5 µm have the greatest probability of deposition in the lung - source. Thus drug particle size is of vital importance in inhaler production.
Other sources:
Pharmacokinetics, pharmacodynamics, and the delivery of pediatric bronchodilator therapy
Lactose information
Methods to identify drug deposition in the lungs following inhalation
The Lungs as a Portal of Entry for Systemic Drug Delivery