Of course, there are exercises that place no stress on the spine and couldn't worsen your scoliosis. Leg extensions using machine, preacher curls, pec flys are some examples of exercises that doesn't involve the spine at all.
Other exercises like shoulder press, deadlifts, squats put pressure and stresses on the spine and might be more risky with certain back conditions. If doctors advise against gym, these might be the exercises that you are supposed to avoid.
I am not sure about the scientific consensus but it is believed (by both doctors and laymen) that scoliosis may lead to uneven pressures that can lead to disc protrusions and some other trauma easier than normally. So you should (as any person) take extreme care and control your back (and whole body) not only during exercise, but also when carrying weights, preparing for exercise and putting weights down after the exercise. This might be the other reason why doctors advise you against visiting the gym.