I would think that connecting the inverter directly to the batteries would make it hard for the charge controller to know what the batteries are really doing. Wouldn't the load on the inverter changing over time as the load on the inverter changes make the charge controller a less effective charger?
