Only cutting half the time is a drawback.
What you're describing is a little more like a treadle lathe.
If you were to put the pottery wheel below the drive shaft and use a belt I would guess that you would get a lot less vibration in the work piece.
Fairly compact treadle lathe
This guy has a great set of videos. If you haven't seen any, you should check them out.