Are you able to style your deer garden as more of a meadow/pasture/brush area? I seed an area each year to cover crops; buckwheat, crimson clover, alfalfa, dikon radish, sunflower, turnips, etc. Ive been doing that for the past couple years, though its starting to self seed which is great. I get 2 main benefits:
1. In the summer I can use it for free ranging chickens, harvest green mulch, and sometimes edible yields.
2. Rather than mow in fall, I leave it standing all winter and the deer have at it, then cut in early spring. I like to think of this area as a peace offering so the deer dont bother my food forest. Occasionally I will also transplant any extra fruit tree or blackberry suckers down in this area to serve as sacrificial decoys.
PS - if you care about the aesthetics, I generally mix in some wildflower seed but thats mainly for my enjoyment, not the deer
