You will not get toxoplasmosis from eating the potatoes, but possibly from any contact with the soil outside, whether in your garden or elsewhere, even if cats are only there occasionally. The pathogen can stay infectious in the soil for a long time. You can get it from other sources too, but I don't know the details. In many places most people have been exposed to it, and may actually be infected without knowing. It is usually not considered a problem (which is good, because it seems extremely difficult to avoid), UNLESS a pregnant woman is infected without any previous exposure - in that case it can be dangerous for the unborn child. Here in Norway I believe pregnant women are routinely tested to see if they have been exposed - if they have not they have to be extra careful about hand and food hygiene, and ideally avoid contact with cats.
So - unless anyone in the household is pregnant, and has previously lived unusually separate from any cats and any contact with soil, toxoplasma is not a big problem. And if it were a problem it would not just be keeping cats out of the potato bed that was an issue.
On the other hand, cat poop in your vegetable beds is of course unappetising, other parasites/infections could be a problem too, and the cat may damage young plants while digging. However, as other people here have already suggested, if the cats have better places to go, you might not have a problem. I usually cover the beds with dead branches or similar material, or even nets or fleece, until the soil has settled and the plants have grown a bit. The cats don't seem interested after that.
Anyway, if you somehow managed to get your next door cat to stay in their house or garden, or managed to get rid of or control any local cats that you know of, you still would not be guaranteed to avoid contact with toxoplasma, in your potato beds, or in any soil in your garden, or anywhere else. Getting one or a few cats locked up, or relocated, or killed, won't really make a difference in that regard.