We have tried to fence them in...
Try fencing them OUT.
As long as there is sufficient food outside the garden, a light duty fence might be enough to keep them out.
Allow them in after harvest: they will clean out the trash and some of the pests & eggs.
Allow them in a week or so before you plant in spring: they will wipe out a lot of the pest larva.
Just keep them out during the growing season.