Could you clarify what you mean by "Open current output" ? If I am not mistaken, you are saying that by adding more
water (opening the 4th nozzle) you are expecting more power, but the power actually decreases?
Where are you measuring current? Is there a meter on the DC side of your rectifier before it enters the battery, or are you getting a reading through a main shunt that is connected to any other circuits? I suspect your charging voltage is a little different now too, since lithium has a much narrower voltage range. If you convert to Watts, does the same effect persist?
I am pretty sure that lithium batteries have quite a bit lower internal resistance than lead-acid, and I am wondering if that could have some effect? Do you have any way to measure the RPMs of the turbine? A cheap bike computer can be tricked into approximating RPM by setting the wheel circumfrence size to 1666mm and then reading it in Km/h. I suppose without knowing what it was doing before, this might not be much help.
Anyway, I also have hydro, but my alternator has a built in rectifier. I am sending high-ish voltage to a charge controller before it goes to the batteries. I am interested in the idea of sending 3 phase AC on my next
project, as I suspect it might help reduce my wire size. I would love to pick your brain at some point.