Of course the down side is that if they're eating everything but the poisonous plants, the poisonous plants will spread.
The more general idea is that animals will eat their preferred foods first, leaving lesser plants to thrive, which is why Polyface Farm and others confine them to a small area at a time, for just long enough that the animals have to eat less preferred plants as well and nothing gets out of balance. Of course that means they have to keep an eye on the "non-preferred" plants not being poisonous, and I don't know how they handle that.
If you're allowing a much less intensive graze, you wouldn't have to keep as close an eye but you might be selecting for poisonous plants on a slower scale. It's of course your option how you want to control them in that case.
