Hi Eli,
That's true, sweet potatoes will be pulling nutrients from the soil. They will also be adding nutrients by the growth and decay of some tubers. Crop rotation does help replenish nutrients in the soil and helps to keep pests populations low. Do you think pests will be an issue?
In our climate sweet potatoes are more of annual. I don't have experience growing sweet potatoes in the same place year after year. So I can't really speak for that. However in our perennial beds, like the asparagus and strawberries, we grow annual crops/flowers to help with the soil. We also add mulch and
compost annually to help keep the soil healthy.