There are two different problems involved in growing potatoes year on year: what happens to the soil and what happens to the potatoes.
You have the obvious problem of nutrient depletion, but this can be solved by fertilization and some soils will have more reserve and take longer to deplete than others. The worse problem is that bacterial and fungal diseases and invertebrate pests will accumulate where potatoes are grown repeatedly. Most of the fungal diseases will not overwinter successfully without a host, but some will. Bacterial diseases like scab get worse and worse as their populations build in the soil and it takes a break of several years to reduce the population.
It is the potato tubers that are the major reserve of disease. It is hard to get them all when you harvest, so you usually end up with some volunteers. These volunteers keep bacterial, fungal, and viral diseases alive through the winter to infect the next crop. If you can get all of the volunteers before you plant a new crop, that significantly reduces the problems of growing in the same ground, but this is a lot of work and requires major soil disruption.
The biggest long term problem is replanting the same tubers. They will accumulate several very common potato viruses which are aphid transmitted and the problem will get worse and worse. Once you have viruses in the population, they will quickly infect even new, clean potatoes that you bring in. You can live with viruses, but they depress yields, often significantly once you accumulate more than one type. The best way to avoid this is to fully harvest your potatoes, eat them all, and plant certified seed each year. Even certified seed is not really virus free, but the prevalence is low enough that the viruses don't become a problem in that year.
If you don't want to be involved in that system of high tech middle-men, then you need to be very attentive to roguing out plants that show any sign of disease and selecting varieties that perform best in your area over multiple years. If you are really serious about growing potatoes, even better would be to breed potatoes for your conditions, growing from seed and keeping the ones that show the best disease resistance. Until about 100 years ago, before the advent of virus testing and micropropagation, this is how cultivars were maintained. New varieties were grown from seed and abandoned when they "ran out" due to accumulation of diseases.
It is amusing to read accounts of potato introductions from the late 19th and early 20th centuries. They go something like:
First year: New potato "Farmer John's Wonder" grown from seed ball.
Third year: Farmer John's Wonder now available in quantity!
Fourth year: Farmer John's Wonder - best potato yield we've ever seen.
Seventh year: Farmer John's Wonder just doesn't perform like it used to. Yields are poor.
Eighth year: "Farmer Bob's Mortgage Lifter" grown from seed ball.
Eleventh year: "Farmer Bob's Mortgage Lifter" - best potato yield we've ever seen.
Most potatoes run out pretty fast, especially if you are growing them where you already have active virus infection. Still, you can often get a solid 3-4 years from seed grown varieties before you retire them.