In most cases, swimming is good exercise. Light swimming is also a therapy for people with joint and back pain. The breaststroke is not usually the recommended stroke for therapy because, if done incorrectly, it can be harmful.
Because you have to keep bringing your head up to breath, the joints in your upper back and neck get extended, and have to deal with a lot of stress from repetitive movement. This can cause a lot pain, especially to those who have experienced back problems before. Having your head up and looking forward can also cause your hips to drop, which puts extra strain on the lower back. This can lead to extra back problems, such as hyperextension.
There are some tips to help you avoid back pain when swimming the backstroke.
Keep your head underwater as long as possible; wearing goggles helps with this
Try to keep your head looking down instead of towards the end of the pool
Get swimming lessons - swimming instructors can be found everywhere, and if you really want to swim the breaststroke without any pain, then it would be a good thing to considers
Overall, there is some middle ground between the things you have heard, but the second thing that you mentioned is probably more accurate. Done correctly, you will be fine, but done incorrectly, you could injure your back.
Swimming and Back Pain