The continued use of chlorinated (and flouridated) waters will indeed have a negative effect on your soil's balance.
Many people counteract this by letting the water sit in barrels or tubs for a day or two.
Chlorine will off-gas rather quickly when exposed to the air. Giving it a good stir periodically
should help speed up the process.