First of all, it's typically liver and the kidneys responsible for removing unwanted chemicals from the body, not the immune system. It should be noted that liver and kidneys have no "knowledge" which chemical is "bad" and which is "good". They have evolved to let some chemicals pass, react some chemicals etc. But there is plethora of compounds that had so little influence on evolutionary process, that we're not ready to handle them, since not being able to handle them was not selected against in the process of evolution.
So can the body break down anything?
tl;dr: No, there are many chemical compounds that are not handled by human body ideally and that, even if there's enough time for the body to handle them, still cause major harm.
Not sure what breaking down exactly means here, but certainly there are some compounds that can get stuck in the body, notably heavy metals. Those and some others accumulate in the body, possibly but not necessarily causing harm in the future. Let's have some examples:
Bioaccumulation
Bioaccumulation is a process where organism (eg. you) absorbs certain chemical compound at faster rate than it excretes or processes it, which is probably what you meant by "breaking down". See bellow some examples of such accumulation. Remember that the compound to be accumulated doesn't need to be exactly toxic, but as the poison is in the dose, as long as you accumulate something, it eventually will became toxic, should it even be water.
Silver
Silver is a heavy metal and was used as disinfection in the first half of 20th century[1]. Silver accumulates in the human body, causing argyria. This often happens to people who adhere to so-called "alternative medicine", which includes promoting silver suspension as a cure to all kinds of diseases.[2]
Lead
Lead is more dangerous than silver and is recognised as important health problem by WHO.
Other examples
I will defer listing of other examples on wikipedia.
Toxification
As I said in the introduction, evolution didn't prepare liver and kidneys for many chemicals. Sometimes the attempt to react them starts the toxic reaction itself and it would be better if the chemical stayed in blood as-is and was excreted in urine. Such event is a type of toxification, toxification can generally occur anywhere, not just liver.
Methanol
Methanol is known as possible toxic component that can accidentally occur in alcoholic beverages. Methanol shares toxicity properties with ethanol, however on top of that, it is metabolized into formic acid in the liver. This is why, if there are no better alternatives, ethanol can be administered to prevent further methanol poisoning since it occupies the liver and therefore stops the toxification process. Bear in mind that this only works before the methanol was already turned into formic acid.[3]
Arsenic
Apart from accumulating, arsenic also is oxidated in body which produces much more toxic arsenic (III) oxide which is the famous deadly poison.
Refferences: